Django with Celery - existing object not found

Question

I am having problem with executing celery task from another celery task.

Here is the problematic snippet (data object already exists in database, its attributes are just updated inside finalize_data function):

def finalize_data(data):
    data = update_statistics(data)
    data.save()
    from apps.datas.tasks import optimize_data
    optimize_data.delay(data.pk)

@shared_task
def optimize_data(data_pk):
    data = Data.objects.get(pk=data_pk)
    #Do something with data

Get call in optimize_data function fails with "Data matching query does not exist."

If I call the retrieve by pk function in finalize_data function it works fine. It also works fine if I delay the celery task call for some time.

This line:

optimize_data.apply_async((data.pk,), countdown=10)

instead of

optimize_data.delay(data.pk)

works fine. But I don't want to use hacks in my code. Is it possible that .save() call is asynchronously blocking access to that row/object?

Answer 1

I know that this is an old post but I stumbled on this problem today. Lee's answer pointed me to the correct direction but I think a better solution exists today.

Using the on_commit handler provided by Django this problem can be solved without a hackish way of countdowns in the code which might not be intuitive to the user about why it exsits.

I'm not sure if this existed when the question was posted but I'm just posting the answer so that people who come here in the future know about the alternative.

Answer 2

I'm guessing your caller is inside a transaction that hasn't committed before celery starts to process the task. Hence celery can't find the record. That is why adding a countdown makes it work.

A 1 second countdown will probably work as well as the 10 second one in your example. I've used 1 second countdowns throughout code to deal with this issue.

Another solution is to stop using transactions.