django loading data from fixture after backward migration / loaddata is using model schema not database schema

Question

I have recenty came across a problem while importing older data than my current model schema. Flow which I use and lead to error:

• dumpdata with python manage.py dumpdata -> 0002
• make some modifications to model
• generate migration with python manage.py schemamigration app_name --auto -> 0003
• run migration
• play with database
• migrate to 0002
• loaddata generate SQL in which I have current (migration 0003) fields, and cause failing loaddata process (mpoly is added field)

  File "/usr/local/lib/python2.6/dist-packages/django/db/backends/postgresql_psycopg2/base.py",

line 44, in execute return self.cursor.execute(query, args) DatabaseError: column "mpoly" of relation "localization_province" does not exist LINE 1: ...e" ("id", "name", "slug", "mpoly") V...

• commenting changes in models.py done before 0003, make all working ok

Is there any way to avoid playing with models after backward migration if I want to loaddata?

Maybe I'm missing something really obvious...

PS: I'm using South 7.3, Django 1.2.3 and PostgreSQL 8.4 as database backend.

Answer 1

Alex Vidal proposed a nice quick fix for this when it bit us at our job. It requires Gary Bernhardt's Dingus library. Once we have time we'll factor out the Dingus dependency and submit a pull request to South, but if you're in a bind right now this may get you out of it:

from dingus import patch


def loaddata(orm, fixture_name):
    _get_model = lambda model_identifier: orm[model_identifier]

    with patch('django.core.serializers.python._get_model', _get_model):
        from django.core.management import call_command
        call_command("loaddata", fixture_name)

Usage:

from apps.common.utils import loaddata


class Migration(DataMigration):
    def forwards(self, orm):
        loaddata(orm, "initial_fjords.json")

We've tested only in Django 1.3 so far. Edit: I checked Django's _get_model history and this should work with Django 0.95 and up.

Answer 2

I find that it is best to keep any fixtures I have in line with the current version of the code. So upon creating migration 0003 you do a data migration and a new dumpdata, replacing fixture 0002. When you create your data migration, make sure you do both forwards and backwards, that way you'll end up with the correct data when you migrate back to 0002.

When you do the data migration, make sure you access all models through the orm object, otherwise you end up with errors similar to what you're already experiencing.

If you want to actually run your django code with the old data (version 0002) for some reason, then your models need to match your database. That would mean checking out an appropriate version of the code using whatever code versioning you're using (git, hg, svn...). If you're trying to solve a problem "back in time" you probably want to branch at that point too.

See also south's documentation comments on fixtures

Here's an idea, inspired by that link above: "the best thing to do is to write a new migration to load the fixture in". What about loading your fixture (which you already have) from a migration, rather than loaddata. You would still need to create a data migration and use the orm object to manually load the data. You can make use of django's serialization functions, which is exactly what loaddata does.

With respect to why loaddata uses the model version and not the database version: loaddata is a django management command, with no knowledge of what south is doing. Because of that, it needs to remain database agnostic and use django's ORM. You can always write your own management command if you need to do something more specific - potentially pulling out some of south's orm versioning, or doing a database specific loaddata that reads the schema directly from the database. I think the solution in the previous paragraph will be a lot less work though.