Circular dependency when squashing Django migrations

Question

We've created a large Django application, and we want to squash migrations. However, the squashed migrations have circular dependencies between the apps in our application. How can we break those circular dependencies without breaking Django's migration squashing?

I've created a small sample project to reproduce the problem. The project has two apps: fruit and meat. An Apple has many Bacon children, and a Bacon has many Cranberry children. You can see that the fruit app depends on the meat app, and the meat app depends on the fruit app.

The first commit creates all three models with a name field on each and foreign keys from Cranberry to Bacon and from Bacon to Apple. Calling makemigrations creates three migrations:

• fruit/0001_initial creates the Apple and Cranberry models
• meat/0001_initial creates the Bacon model with its foreign key to Apple
• fruit/0002_cranberry_bacon adds the foreign key from Cranberry to Bacon

The next commit adds an Apple.size field just so there is something to squash. Calling makemigrations adds another migration:

• fruit/0003_apple_size adds the size field

Running squashmigrations now creates a squashed migration with a circular dependency. The squashmigrations documentation gives this advice:

To manually resolve a CircularDependencyError, break out one of the ForeignKeys in the circular dependency loop into a separate migration, and move the dependency on the other app with it. If you’re unsure, see how makemigrations deals with the problem when asked to create brand new migrations from your models. In a future release of Django, squashmigrations will be updated to attempt to resolve these errors itself.

If I do that, however, the extra migration isn't configured properly as a replacement. That means that my current database that has gone through the original migrations tries to add the foreign key field again and fails.

$ ./manage.py migrate
...
django.db.utils.ProgrammingError: column "bacon_id" of relation "fruit_cranberry" already exists

How can I tell the migration system that two new migrations replace all the old migrations?

Answer 1

This seems like a lot of work, but it's the best solution I've found so far. I've posted the squashed migrations in the master branch. Before running squashmigrations, we replace the foreign key from Cranberry to Bacon with an integer field. Override the field name so it has the _id suffix of a foreign key. This will break the dependency without losing data.

# TODO: switch back to the foreign key.
# bacon = models.ForeignKey('meat.Bacon', null=True)
bacon = models.IntegerField(db_column='bacon_id', null=True)

Run makemigrations and rename the migration to show that it is starting the squash process:

• fruit/0100_unlink_apps converts the foreign key to an integer field

Now run squashmigrations fruit 0100 and rename the migration to make it easier to follow the sequence:

• fruit/0101_squashed combines all the migrations from 1 to 100.

Comment out the dependency from fruit/0101_squashed to meat/0001_initial. It isn't really needed, and it creates a circular dependency. With more complicated migration histories, the foreign keys to other apps might not get optimized out. Search the file for all the app names listed in the dependencies to see if there are any foreign keys left. If so, manually replace them with the integer fields. Usually, this means replacing a CreateModel(...ForeignKey...) and AlterModel(...IntegerField...) with a CreateModel(...IntegerField...).

The next commit contains all these changes for demonstration purposes. It wouldn't make sense to push it without the following commit, though, because the apps are still unlinked.

Switch back to the foreign key from Cranberry to Bacon, and run makemigrations one last time. Rename the migration to show that it is finishing the squash process:

• fruit/0102_relink_apps converts the integer field back to a foreign key

Remove the dependency from fruit/0102_relink_apps to fruit/0101_squashed, and add a dependency from fruit/0102_relink_apps to fruit/0100_unlink_apps. The original dependency just won't work. Take the dependencies that were commented out in fruit/0101_squashed and add them to fruit/0102_relink_apps. That will ensure the links get created in the right order.

Run the test suite to show that the squashed migration works properly. If you can, test against something other than SQLite, because it doesn't catch some foreign key problems. Back up the development or production database and run migrate to see that the unlinking and relinking of the apps doesn't break anything.

Take a nap.

Bonus section: after all installations are squashed

The convert_squash branch shows what could happen in the future once all installations have migrated past the squash point. Delete all the migrations from 1 to 100, because they've been replaced by 101. Delete the replaces list from fruit/0101_squashed. Run showmigrations to check for any broken dependencies, and replace them with fruit/0101_squashed.

The horror of many-to-many relationships

If you are unlucky enough to have a many-to-many relationship between two apps, it gets really ugly. I had to use the SeparateDatabaseAndState operation to disconnect the two apps without having to write a data migration. The trick is to replace the many-to-many relationship with a temporary child model using the same table and field names, then tell Django to just update its state without touching the database schema. To see an example, look at my unlink, squashed, and relink migrations.