Django data migration fails when running manage.py test, but not when running manage.py migrate

Question

I have a Django 1.7 migration that looks something like this:

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations

def units_to_m2m(apps, schema_editor):
    Interval = apps.get_model("myapp", "Interval")
    IntervalUnit = apps.get_model("myapp", "IntervalUnit")

    for interval in Interval.objects.all():
        IntervalUnit(
            interval=interval,
            unit=interval.unit,
            base_date=interval.base_date
        ).save()

class Migration(migrations.Migration):

    dependencies = [
        ('otherapp', '0007_auto_20150310_1400'),
        ('myapp', '0009_auto_20150316_1608'),
    ]

    operations = [
        migrations.CreateModel(
            name='IntervalUnit',
            # ...
        ),
        # ...
        migrations.AddField(
            model_name='interval',
            name='units',
            field=models.ManyToManyField(to='otherapp.Unit', through='myapp.IntervalUnit'),
            preserve_default=True,
        ),
        migrations.RunPython(units_to_m2m),
        migrations.RemoveField(
            model_name='interval',
            name='unit',
        ),
        migrations.RemoveField(
            model_name='interval',
            name='base_date',
        ),
    ]

When I run manage.py migrate, it migrates just fine. When I run manage.py test, however, it tries to create the test database, then fails in the middle of this migration with the following error:

Traceback (most recent call last):
...
  File "/home/adam/myproject/myapp/migrations/0010_auto_20150317_1516.py", line 10, in units_to_m2m
    for interval in Interval.objects.all():
...
django.db.utils.OperationalError: (1054, "Unknown column 'myapp_interval.base_date' in 'field list'")

When I connect to the test database afterwards (it doesn't delete it), the database structure looks as you'd expect after the migration has run, even though it crashed half way through. What's going on here?

Edit: I've tried splitting the migration into three separate migrations, one containing all the stuff before the RunPython one, one containing RunPython on its own, and one containing all the stuff that's afterwards; it's still doing the exact same thing.

Answer 1

This is kind of weird and we don't know why this works, but we changed our allow_migrate signature in our router to the following:

def allow_migrate(self, db, app_label, **hints):
    """
    Make sure the mydb db does not allow migrations
    """
    if db == 'mydb':
        return False

    return True

And this error mysteriously went away. Note that this signature does not match what is in the 1.8 documentation (we're using 1.8.2) allow_migrate(db, app_label, model_name=None, **hints) as shown here: https://docs.djangoproject.com/en/1.8/topics/db/multi-db/#allow_migrate

But hopefully this will help you?

Answer 2

It turns out, the migrations were running successfully, in the order they were supposed to, but I have two databases, and it was running my migrations on both without consulting the database router. The Django ticket to track this problem is #23273, which is still open.

Presumably, the RunPython migration was querying against default (which had already been migrated) rather than the database the migration was actually supposed to run on.

In my case, we no longer needed to use the second database for anything, so we were able to remove it from settings.DATABASES altogether.