Adding a non-null ForeignKey field in Django+South

Question

I use Django and South for my database. Now I want to add a new Model and a field in an existing model, referencing the new model. For example:

class NewModel(models.Model):
    # a new model
    # ... 

class ExistingModel(models.Model):
    # ... existing fields

    new_field = models.ForeignKey(NewModel)  # adding this now

Now South obviously complains that I added a non-null field and asks me to enter a one-off value. But what I really want is to create a new NewModel instance for every existing ExistingModel instance, thus fulfilling the database requirements. Is that possible somehow?

Answer 1

The easiest way to do this is to write a schema migration that makes the column change, and then write a datamigration to correctly fill in the value. Depending on the database you're using you'll have to do this in slightly different ways.

Sqlite

For Sqlite, you can add a sentinel value for the relation and use a datamigration to fill it in without any issue:

0001_schema_migration_add_foreign_key_to_new_model_from_existing_model.py

from south.db import db
from south.v2 import SchemaMigration
from django.db import models

class Migration(SchemaMigration):
    
    def forwards(self, orm):
        db.add_column('existing_model_table', 'new_model',
                      self.gf('django.db.models.fields.related.ForeignKey')(default=0, to=orm['appname.new_model']), keep_default=False)

0002_data_migration_for_new_model.py:

class Migration(DataMigration):
    
    def forwards(self, orm):
        for m in orm['appname.existing_model'].objects.all():
            m.new_model = #custom criteria here
            m.save()
    

This will work just fine, with no issues.

Postgres and MySQL

With MySql, you have to give it a valid default. If 0 isn't actually a valid Foreignkey, you'll get errors telling you so.

You could default to 1, but there are instances where that isn't a valid foreign key (happened to me because we have different environments, and some environments publish to other databases, so the IDs rarely match up (we use UUIDs for cross-database identification, as God intended).

The second issue you get is that South and MySQL don't play well together. Partially because MySQL doesn't have the concept of DDL transactions.

In order to get around some issues you will inevitably face (including the error I mentioned above and from South asking you to mark orm items in a SchemaMigration as no-dry-run), you need to change the above 0001 script to do the following:

0001_schema_migration_add_foreign_key_to_new_model_from_existing_model.py

from south.db import db
from south.v2 import SchemaMigration
from django.db import models

class Migration(SchemaMigration):
    
    def forwards(self, orm):
        id = 0
        if not db.dry_run:
            new_model = orm['appname.new_model'].objects.all()[0]
            if new_model:
                id = new_model.id
        db.add_column('existing_model_table', 'new_model',
                      self.gf('django.db.models.fields.related.ForeignKey')(default=id, to=orm['appname.new_model']), keep_default=False)

And then you can run the 0002_data_migration_for_new_model.py file as normal.

I advise using the same example above for Postgres and for MySql. I don't remember any issues offhand with Postgres with the first example, but I'm certain the second example works for both (tested).

Answer 2

You want a data migration to supplement your schema migration in this scenario.

South has a nice step by step tutorial on how to achieve this in the docs, here.

It's not uncommon in South to have the desired outcome spread over two or three schema/data migrations as its not always possible to do it in one big hit (sometimes depends on the underlying db if it will tolerate adding a non null column with no default). So in this case you might add a schema migration that has a default, then a data migration with your object manipulation then a final schema migration.