django migrations - workflow with multiple dev branches

Question

I'm curious how other django developers manage multiple code branches (in git for instance) with migrations.

My problem is as follows: - we have multiple feature branches in git, some of them with django migrations (some of them altering fields, or removing them altogether) - when I switch branches (with git checkout some_other_branch) the database does not reflect always the new code, so I run into "random" errors, where a db table column does not exist anymore, etc...

Right now, I simply drop the db and recreate it, but it means I have to recreate a bunch of dummy data to restart work. I can use fixtures, but it requires keeping track of what data goes where, it's a bit of a hassle.

Is there a good/clean way of dealing with this use-case? I'm thinking a post-checkout git hook script could run the necessary migrations, but I don't even know if migration rollbacks are at all possible.

Answer 1

Migrations rollback are possible and usually handled automatically by django.

Considering the following model:

class MyModel(models.Model):
    pass
    

If you run python manage.py makemigrations myapp, it will generate the initial migration script. You can then run python manage.py migrate myapp 0001 to apply this initial migration.

If after that you add a field to your model:

class MyModel(models.Model):    
    my_field = models.CharField()
    

Then regenerate a new migration, and apply it, you can still go back to the initial state. Just run python manage.py migrate myapp 0001 and the ORM will go backward, removing the new field.

It's more tricky when you deal with data migrations, because you have to write the forward and backward code. Considering an empty migration created via python manage.py makemigrations myapp --empty, you'll end up with something like:

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations

def forward(apps, schema_editor):
    # load some data
    MyModel = apps.get_model('myapp', 'MyModel')
    
    while condition:
        instance = MyModel()
        instance.save()
    
def backward(apps, schema_editor):
    # delete previously loaded data
    MyModel = apps.get_model('myapp', 'MyModel')
    
    while condition:
        instance = MyModel.objects.get(myargs)
        instance.delete()

class Migration(migrations.Migration):

    dependencies = [
        ('myapp', '0003_auto_20150918_1153'),
    ]

    operations = [ 
        migrations.RunPython(forward, backward),
    ]
    

For pure data-loading migrations, you usually don't need the backward migration. But when you alter the schema and update existing rows (like converting all values in a column to slug), you'll generally have to write the backward step.

In our team, we try to avoid working on the same models at the same time to avoid collision. If it is not possible, and two migration with the same number (e.g 0002) are created, you can still rename one of them to change the order in which they will be applied (also remember to update the dependencies attribute on the migration class to your new order).

If you end up working on the same model fields at the same time in different features, you'll still be in trouble, but it may mean these features are related and should be handled together in a single branch.

For the git-hooks part, it's probably possible to write something, Assuming your are on branch mybranch and want to check out another feature branch myfeature:

1. Just before switching, you dump the list of currently applied migrations into a temporary file mybranch_database_state.txt
2. Then, you apply myfeature branch migrations, if any
3. Then, when checking back mybranch, you reapply your previous database state by looking to the dump file.

However, it seems a bit hackish to me, and it would probably be really difficult to handle properly all scenarios: rebasing, merging, cherry-picking, etc.

Handling the migrations conflicts when they occurs seems easier to me.

Answer 2

I don't have a good solution to this, but I feel the pain.

A post-checkout hook will be too late. If you are on branch A and you check out branch B, and B has fewer migrations than A, the rollback information is only in A and needs to be run before checkout.

I hit this problem when jumping between several commits trying to locate the origin of a bug. Our database (even in development trim) is huge, so dropping and recreating isn't practical.

I'm imagining a wrapper for git-checkout that:

1. Notes the newest migration for each of your INSTALLED_APPS
2. Looks in the requested branch and notes the newest migrations there
3. For each app where the migrations in #1 are farther ahead than in #2, migrate back to the highest migration in #2
4. Check out the new branch
5. For each app where migrations in #2 were ahead of #1, migrate forward

A simple matter of programming!