Celery Tasks Lost after sending to rabbitmq broker

Question

I am having a strange issue where some of my tasks are getting dropped after sent to broker. This happens for around 1 out of 10 tasks. I have checked that there is no old celery worker consuming the task.

I used database backed backend and flower to monitor the missing tasks, but the task_id returned after apply_async is not present in database or in flower. Its status always comes as pending.

Then I used celery signals to find out what's happening. I found that for the missing tasks, only before_task_publish and after_task_publish signals are fired. Post that there is no trace for this task.

These are my signals

@before_task_publish.connect
def before_task_publish_handler(sender=None, headers=None, body=None, **kwargs):
# information about task are located in headers for task messages
# using the task protocol version 2.
  logger.info("BEFORE TASK SENT  id:"+body['id'])



@after_task_publish.connect
def after_task_publish_handler(sender=None, headers=None, body=None, exchange=None, routing_key=None, **kwargs):
# information about task are located in headers for task messages
# using the task protocol version 2.
    logger.info("AFTER TASK SENT  id:"+body['id'])


@task_prerun.connect
def task_prerun_handler(sender=None, task_id=None, task=None, **kwargs):
  logger.info("TASK PRERUN with TASK_ID:"+str(task_id))

This is what I found in logs

$ cat gunicorn-access.log | grep -i 103de274-00dc-4765-844f-d319e9e199c2
  BEFORE TASK SENT id: '103de274-00dc-4765-844f-d319e9e199c2'
  AFTER TASK SENT  id: '103de274-00dc-4765-844f-d319e9e199c2'

I am not sure whether the task is ignored by rabbitmq or its silently dropped for some reason.

Answer 1

Once in a while celery loses tasks before it gets executed. If you don't want to lose them, you should set enable task_acks_late(CELERY_ACKS_LATE in old versions).

In your celery settings, set

task_acks_late = True

This makes sure that the task messages will be acknowledged after the task has been executed.