Django hangs on applying huge migration

Question

I have a table in Postgres 9.5 with about 7KK rows. Django version is 1.10.5. Database and App are different servers with Ubuntu 16.04.2 within one local network.

Django's gunicorn server is stopped, so no other operations are being executed. I'm going to add one field there:

migrations.AlterField(
            model_name='balanceentry',
            name='reason',
            field=models.CharField(
                choices=[(b'default', b'Default'), (b'referral', b'Referral'), (b'referrer', b'Referrer'),
                         (b'random', b'Random'), (b'android_offer', b'Android Offer'), (b'ios_offer', b'iOS Offer'),
                         (b'offerwall', b'Offerwall'), (b'withdrawal', b'Withdrawal')],
                default=b'default', db_index=True, max_length=32),
        ),

And then I'm applying it:

$ ./manage.py migrate users 0026_auto_20170419_1758
Operations to perform:
  Target specific migration: 0026_auto_20170419_1758, from users
Running migrations:
  Applying users.0026_auto_20170419_1758...

And monitoring Postgres with pg_top.

It's doing ALTER for about 15 minutes, then I see this:

34567 postgres  20    0  401M   39M sleep   2:17  0.10%  0.00% postgres: *** *** ip(45200) idle in transaction

This does not change for about 10 minutes (only WCPU is changing from 0% to about 0.1% and back).

Then this record disappears (I think this means that client is disconnected), but ./manage.py migrate ... does not change its status at all, it just stays "running" with no changes (I've waited for about 2 hours).

I tried to restart postgres service, if I do this, it rolls back transaction (I think), frees some disk space, but migrate management command still hangs. It does not even react on Ctrl-C, I can only kill it with -9.

I also noticed that about 1-2 GB free disk space disappears after every attempt.

So, how can I fix this issue?

Answer 1

I am unable to make comment, because of too little reputation, so it will have to be an answer.

I do not have a solution, but an idea to a workaround. You can type python manage.py sqlmigrate which will show the SQL it performs, and then manually run it towards the database. If this can succeed, you need to add an entry in the database-table django_migrations with the migration name and app for the migration.

I have no idea if this will work, but you might be able to pinpoint the error in the SQL statement and optimize it a bit.