Alter model to add "through" relationship to order a ManytoMany field - Django 1.7 migration modification

Question

I am trying to add an order to a ManyToMany field that I created a while ago. I basically want to order pictures in collections of pictures. I am running on Django 1.7, so no more South migrations (I was trying to follow this tutorial: http://mounirmesselmeni.github.io/2013/07/28/migrate-django-manytomany-field-to-manytomany-through-with-south/)

Here's the "through" relationship that I have:

class CollectionPictures(models.Model):
    picture = models.ForeignKey(
        Picture,
        verbose_name=u'Picture',
        help_text=u'Picture is included in this collection.',
    )
    collection = models.ForeignKey(
        Collection,
        verbose_name=u'Collection',
        help_text=u'Picture is included in this collection',
    )
    order = models.IntegerField(
        verbose_name=u'Order',
        help_text=u'What order to display this picture within the collection.',
        max_length=255
    )

    class Meta:
        verbose_name = u"Collection Picture"
        verbose_name_plural = u"Collection Pictures"
        ordering = ['order', ]

    def __unicode__(self):
        return self.picture.name + " is displayed in " + self.collection.name + (
        " in position %d" % self.order)


class Collection(models.Model):
    pictures = models.ManyToManyField(Picture, through='CollectionPictures', null=True)
    [... Bunch of irrelevant stuff after]

So this should work if I didn't have to migrate my old data (the only difference in the model is that it didn't have the through='CollectionPictures'

Here's my migration :

class Migration(migrations.Migration):

    dependencies = [
        ('artist', '0002_auto_20141013_1451'),
        ('business', '0001_initial'),
    ]

    operations = [
        migrations.CreateModel(
            name='CollectionPictures',
            fields=[
                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                ('order', models.IntegerField(help_text='What order to display this picture within the collection.', max_length=255, verbose_name='Order')),
                ('collection', models.ForeignKey(verbose_name='Collection', to='business.Collection', help_text='Picture is included in this collection')),
                ('picture', models.ForeignKey(verbose_name='Picture', to='artist.Picture', help_text='Picture is included in this collection.')),
            ],
            options={
                'ordering': ['order'],
                'verbose_name': 'Collection Picture',
                'verbose_name_plural': 'Collection Pictures',
            },
            bases=(models.Model,),
        ),
        migrations.AlterField(
            model_name='collection',
            name='pictures',
            field=models.ManyToManyField(to=b'artist.Picture', null=True, through='business.CollectionPictures'),
        ),
    ]

This throws an error when migrating:

ValueError: Cannot alter field business.Collection.pictures into business.Collection.pictures - they are not compatible types (you cannot alter to or from M2M fields, or add or remove through= on M2M fields)

Has anybody already tried that kind of manipulation with the new 1.7 migrations?

Thanks !

Answer 1

The safest approach would be to create a new field and copy the data over.

1. Leave pictures alone and add pictures2 with your through field. Run makemigrations.

2. Edit the generated migration file and add a RunPython command where you copy data from the old table to the new table. Perhaps you can programmatically choose a good value for the new order column as well.

3. Delete the old pictures field. Run makemgirations.

4. Rename pictures2 to pictures. Run makemigrations.

This approach should leave you in the state you want with your data intact.

If copying over the data is a big problem you could try something else, like adding the order column in SQL, using the db_table option on CollectionPictures to make it point to the existing table, and then wiping out migrations and redoing with --fake. But that seems riskier than the approach above.

Answer 2

Old question, but I had this problem too, and I've found a way with Django 1.11 that works, and should work with older versions too. The needed class exists back to 1.7 and still exists in 2.0

The fix involves manually changing the migration to do what we want, using the SeparateDatabaseAndState migration class. This class lets Django update the state, but gives us control over what operations to perform. In this case we just want to rename the model table, everything else is already set up right.

The steps:

1. Create your new ManyToMany Through model, but specify a custom table name, and no extra fields:

   class CollectionPictures(models.Model):
       collection = ...
       picture = ...
       class Meta:
           # Change myapp to match.
           db_table = "myapp_collection_pictures"
           unique_together = (("collection", "picture"))
   

2. Taking the existing migration, and take the operations it generates and wrap it all in a single new SeparateDatabaseAndState:

   class Migration(migrations.Migration):
   
       dependencies = [
           ('artist', '0002_auto_20141013_1451'),
           ('business', '0001_initial'),
       ]
   
       operations = [
           migrations.SeparateDatabaseAndState(
               database_operations=[
               ],
               state_operations=[
                   migrations.CreateModel(
                       name='CollectionPictures',
                       fields=[
                           ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                           ('order', models.IntegerField(help_text='What order to display this picture within the collection.', max_length=255, verbose_name='Order')),
                           ('collection', models.ForeignKey(verbose_name='Collection', to='business.Collection', help_text='Picture is included in this collection')),
                           ('picture', models.ForeignKey(verbose_name='Picture', to='artist.Picture', help_text='Picture is included in this collection.')),
                       ],
                       options={
                           'ordering': ['order'],
                           'verbose_name': 'Collection Picture',
                           'verbose_name_plural': 'Collection Pictures',
                       },
                       bases=(models.Model,),
                   ),
                   migrations.AlterField(
                       model_name='collection',
                       name='pictures',
                       field=models.ManyToManyField(to=b'artist.Picture', null=True, through='business.CollectionPictures'),
                   ),
               ]
           )
   

3. Remove the db_table from the class Meta, and add this operation after the SeparateDatabaseAndState, (not into the database_operations.):

   migrations.AlterModelTable(
       name='collectionpicture',
       table=None,
   ),
   

Now if you run `./mange.py sqlmigrate myapp 0003 (pick the right number prefix!) you should with any luck see something like this as output:

BEGIN;
--
-- Custom state/database change combination
--
--
-- Rename table for collection[Pictures to None
--
ALTER TABLE "myapp_collection_pictures" RENAME TO "myapp_collectionpictures";
COMMIT;

4. Add your new columns ("order" in this case) and create a new migration. It's probably possible to do this at the same time, but I decided it was easier to do in two migrations.

(Step 3 isn't strictly required if you are happy keeping the custom table name there.)

And double check with ./manage.py makemigrations --check -- it should print "No changes detected".