It's possible to perform db operations asynchronously in django?

Question

I'm writing a command to randomly create 5M orders in a database.

def constrained_sum_sample(
    number_of_integers: int, total: Optional[int] = 5000000
) -> int:
    """Return a randomly chosen list of n positive integers summing to total.

    Args:
        number_of_integers (int): The number of integers;
        total (Optional[int]): The total sum. Defaults to 5000000.

    Yields:
        (int): The integers whose the sum is equals to total.
    """

    dividers = sorted(sample(range(1, total), number_of_integers - 1))
    for i, j in zip(dividers + [total], [0] + dividers):
        yield i - j


def create_orders():
    customers = Customer.objects.all()
    number_of_customers = Customer.objects.count()
    for customer, number_of_orders in zip(
        customers,
        constrained_sum_sample(number_of_integers=number_of_customers),
    ):
        for _ in range(number_of_orders):
            create_order(customer=customer)

number_of_customers will be at least greater than 1k and the create_order function does at least 5 db operations (one to create the order, one to randomly get the order's store, one to create the order item (and this can go up to 30, also randomly), one to get the item's product (or higher but equals to the item) and one to create the sales note.

As you may suspect this take a LONG time to complete. I've tried, unsuccessfully, to perform these operations asynchronously. All of my attempts (dozen at least; most of them using sync_to_async) have raised the following error:

SynchronousOnlyOperation you cannot call this from an async context - use a thread or sync_to_async

Before I continue to break my head, I ask: is it possible to achieve what I desire? If so, how should I proceed?

Thank you very much!

Answer 1

Not yet supported but in development.

Django 3.1 has officially asynchronous support for views and middleware however if you try to call ORM within async function you will get SynchronousOnlyOperation.

if you need to call DB from async function they have provided helpers utils like: async_to_sync and sync_to_async to change between threaded or coroutine mode as follows:

from asgiref.sync import sync_to_async

results = await sync_to_async(Blog.objects.get, thread_sensitive=True)(pk=123)

If you need to queue call to DB, we used to use tasks queues like celery or rabbitMQ.

• By the way if you really know what you are doing you can call it but on your responsibility just turn off the Async safety but watch out for data lost and integrity errors

#settings.py
DJANGO_ALLOW_ASYNC_UNSAFE=True

The reason this is needed in Django is that many libraries, specifically database adapters, require that they are accessed in the same thread that they were created in. Also a lot of existing Django code assumes it all runs in the same thread, e.g. middleware adding things to a request for later use in views.

More fun news in the release notes: https://docs.djangoproject.com/en/3.1/topics/async/

Answer 2

It's possible to achieve what you desire, however you need a different perspective to solve this problem.

Try using asynchronous workers, and a simple one would be rq workers or celery.

Use one of these libraries to process async long-running tasks defined in django in different threads or processes.