Django celery task: Newly created model DoesNotExist

Question

Why is a model instance I've created, when queried from a celery task started directly afterwards, not found? For example:

# app.views

model = Model.objects.create()    # I create my lovely model in a view
from app.tasks import ModelTask   # I import my Async celery task
ModelTask.delay(model.pk)         # I start the task

That all looks fine, and surely if I queried at any point after the create() call the model should exist in the database.

Update 1: I'm using the default transaction.autocommit behaviour, that Django provides, for my view.

But the task below throws an ObjectDoesNotExist exception:

# app.tasks

class ModelTask(Task):
    def run(self, model_pk):
        from app.models import Model
        Model.objects.get(pk=model_pk)

In my tests, as expected, model_pk is a correct positive integer ID.

Conclusion

I assume there is some asynchronous/"separate process" issues arising here, but I don't know what it is. If feel as though there is some obvious mistake I'm making.

I don't think that database transactions are the answer, because Django's default "autocommit" approach ensures DB actions are performed as soon as the create() method is called.

Answer 1

These answers need to be updated. Django now has transaction.on_commit() which is built for this exact problem, they even provide an example with a task:

transaction.on_commit(lambda: some_celery_task.delay('arg1'))

https://docs.djangoproject.com/en/2.1/topics/db/transactions/#django.db.transaction.on_commit

Answer 2

I've had the same problem in my code. After long investigation, I found out that race condition was happening because I was using @transaction.commit_on_success decorator. So the transaction was committed only after view returned. Which was happening after I was calling celery task.

Once I've removed "commit_on_success" decorator, everything started to work as expected. Because Django's default transaction behavior is to commit transaction after any database altering operation.

You might also want to make sure you are not using TransactionMiddleware, because it does similar thing to @transaction.commit_on_success decorator. If you want to keep using it, you should look into using @transaction.autocommit decorator in your views with celery tasks, or @transaction.commit_manually.