Django: RunSQL: use PostgreSQL COPY command

Question

I try to run a migration with the following RunSQL command:

class Migration(migrations.Migration):
    operations = [
        RunSQL(
r'''
COPY auth_group (id, name) FROM stdin;
1   TEST-GROUP
\.
''')]

It fails like this:

File "/home/foo/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 63, in execute
     return self.cursor.execute(sql)
django.db.utils.ProgrammingError: syntax error at or near "1"
LINE 3: 1 TEST-GROUP

Is there COPY not allowed in RunSQL?

We use psycopg2

Answer 1

The psycopg2 driver exposes copy_to and copy_from methods that can be used to implement the same behavior as the psql client; the key is to use the RunPython operation instead of the RunSQL operation.

You'll need:

• A function in your migration to open your file and interact with the copy methods
• A RunPython operation in your migration's operations list to invoke the function

Example, using Django 1.8.4, Python 2.7.10, psycopg2 2.6.1 -

from django.db import models, migrations

def forwards(apps, schema_editor):
    with open('./path/to/file.csv', 'r') as infile:
        with schema_editor.connection.cursor() as stmt:
            #for finer control, use copy_expert instead
            stmt.copy_from(infile, 'your_table', sep=',')

class Migration(migrations.Migration):

    dependencies = [
    ]

    operations = [
        #this runs your function
        migrations.RunPython(forwards)
    ]

Some notes:

• The file object passed to copy is essentially STDIN in the statement.
• The copy command can be picky about columns; using copy_expert you can control all the same options as the command: format, headers, delimiter, etc.

For more info on the copy_* methods, check the psycopg2 docs: http://initd.org/psycopg/docs/cursor.html