pytest-django database initialization doesn't appear to reload the database

Question

We have distilled a situation down to the following:

import pytest
from django.core.management import call_command
from foo import bar

@pytest.fixture(scope='session')
def django_db_setup(django_db_setup, django_db_blocker):
    LOGGER.info('ran call_command')
    with django_db_blocker.unblock():
        call_command('loaddata', 'XXX.json')

@pytest.mark.django_db(transaction=True)
def test_t1():
    assert len(bar.objects.all())

@pytest.mark.django_db(transaction=True)
def test_t2():
    assert len(bar.objects.all())

The test fixture XXX.json includes one bar. The first test (test_t1) succeeds. The second test (test_t2) fails. It appears that the transaction=True attribute does not result in the database being reinitialized with the data from the test fixture.

If TransactionTestCase from unittest is used instead, the initialization happens before every test case in the class and all tests succeed.

from django.test import TransactionTestCase

from foo import bar

class TestOne(TransactionTestCase):

    fixtures = ['XXX.json']

    def test_tc1(self):
        assert len(bar.objects.all())

    def test_tc2(self):
        assert len(bar.objects.all())
        objs = bar.objects.all()
        for bar in objs:
            bar.delete()

    def test_tc3(self):
        assert len(bar.objects.all())

I would appreciate any perspectives on why the pytest example doesn't result in a reinitialized database for the second test case.

Answer 1

The django_db_setup is session scoped, and therefore only run once at the beginning of the test session. When using transaction=True, the database gets flushed after every test (including the first) and so any data added in django_db_setup is removed.

TransactionTestCase obviously knows that it is using transactions and because it is a django thing it knows that it needs to re-add the fixtures for each test, but pytest in general is not aware of django's needs, and so it has no way to know that it needs to re-run your fixture django_db_setup – as far as it's concerned it only needs to run that once since it is session scoped.

You have the following options:

1. use a lower scoped fixture, probably to the function scope as suggested in the comments. But this will probably be opt-in, and this will be run within the transaction, so will be removed after the test is complete.
2. Write a fixture that is smart / django-aware, and knows when it needs to re-populate that data by detecting when the test is using transactions. But you need to ensure that the database connection being used is not in a transaction. I have done this on django 1.11 and it works fine, although it may need fixing after an upgrade. Looks something like this:

from unittest.mock import patch

from django.core.management import call_command
from django.db import DEFAULT_DB_ALIAS, ConnectionHandler

import pytest


_need_data_load = True


@pytest.fixture(autouse=True)
def auto_loaddata(django_db_blocker, request):
    global _need_data_load
    if _need_data_load:
        # Use a separate DB connection to ensure we're not in a transaction.
        con_h = ConnectionHandler()
        try:
            def_con = con_h[DEFAULT_DB_ALIAS]
            # we still need to unblock the database because that's a test level
            # constraint which simply monkey patches the database access methods
            # in django to prevent access.
            # 
            # Also note here we need to use the correct connection object
            # rather than any default, and so I'm assuming the command
            # imports `from django.db import connection` so I can swap it.
            with django_db_blocker.unblock(), patch(
                'path.to.your.command.modules.connection', def_con
            ):
                call_command('loaddata')
        finally:
            con_h.close_all()
        _need_auto_sql = False

    using_transactional_db = (
        'transactional_db' in request.fixturenames
        or 'live_server' in request.fixturenames
    )
    if using_transactional_db:
        # if we're using a transactional db then we will dump the whole thing
        # on teardown, so need to flag that we should set it up again after.
        _need_data_load = True