"django.db.utils.ProgrammingError: relation "app_user" does not exist" during manage.py test

Question

My setup:

• Django 1.8.3
• Python 2.7.10
• Ubuntu 14.04
• django-two-factor-auth==1.2.0

I get the following error when I run python manage.py test:

Traceback (most recent call last):
  File "/src/venv/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "/src/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "/src/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 330, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/commands/test.py", line 30, in run_from_argv
    super(Command, self).run_from_argv(argv)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/base.py", line 393, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/commands/test.py", line 74, in execute
    super(Command, self).execute(*args, **options)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/base.py", line 444, in execute
    output = self.handle(*args, **options)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/commands/test.py", line 90, in handle
    failures = test_runner.run_tests(test_labels)
  File "/src/venv/lib/python2.7/site-packages/django/test/runner.py", line 210, in run_tests
    old_config = self.setup_databases()
  File "/src/venv/lib/python2.7/site-packages/django/test/runner.py", line 166, in setup_databases
    **kwargs
  File "/src/venv/lib/python2.7/site-packages/django/test/runner.py", line 370, in setup_databases
    serialize=connection.settings_dict.get("TEST", {}).get("SERIALIZE", True),
  File "/src/venv/lib/python2.7/site-packages/django/db/backends/base/creation.py", line 368, in create_test_db
    test_flush=not keepdb,
  File "/src/venv/lib/python2.7/site-packages/django/core/management/__init__.py", line 120, in call_command
    return command.execute(*args, **defaults)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/base.py", line 444, in execute
    output = self.handle(*args, **options)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 179, in handle
    created_models = self.sync_apps(connection, executor.loader.unmigrated_apps)
  File "/src/venv/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 317, in sync_apps
    cursor.execute(statement)
  File "/src/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/src/venv/lib/python2.7/site-packages/django/db/utils.py", line 97, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/src/venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 63, in execute
    return self.cursor.execute(sql)
django.db.utils.ProgrammingError: relation "app_user" does not exist

When I drop a print(sql) statement on line 62 in django/db/backends/utils.py, I get following output:

CREATE DATABASE "test_dev"

            SELECT c.relname, c.relkind
            FROM pg_catalog.pg_class c
            LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
            WHERE c.relkind IN ('r', 'v')
                AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
                AND pg_catalog.pg_table_is_visible(c.oid)
CREATE TABLE "django_migrations" ("id" serial NOT NULL PRIMARY KEY, "app" varchar(255) NOT NULL, "name" varchar(255) NOT NULL, "applied" timestamp with time zone NOT NULL)

            SELECT c.relname, c.relkind
            FROM pg_catalog.pg_class c
            LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
            WHERE c.relkind IN ('r', 'v')
                AND n.nspname NOT IN ('pg_catalog', 'pg_toast')
                AND pg_catalog.pg_table_is_visible(c.oid)
SAVEPOINT "s140275211773760_x1"
CREATE TABLE "distributedlock_lock" ("id" serial NOT NULL PRIMARY KEY, "key" varchar(255) NOT NULL, "value" varchar(255) NOT NULL, "timestamp" timestamp with time zone NULL)
RELEASE SAVEPOINT "s140275211773760_x1"
SAVEPOINT "s140275211773760_x2"
CREATE TABLE "djkombu_queue" ("id" serial NOT NULL PRIMARY KEY, "name" varchar(200) NOT NULL UNIQUE)
RELEASE SAVEPOINT "s140275211773760_x2"
SAVEPOINT "s140275211773760_x3"
CREATE TABLE "djkombu_message" ("id" serial NOT NULL PRIMARY KEY, "visible" boolean NOT NULL, "sent_at" timestamp with time zone NULL, "payload" text NOT NULL, "queue_id" integer NOT NULL)
RELEASE SAVEPOINT "s140275211773760_x3"
SAVEPOINT "s140275211773760_x4"
CREATE TABLE "otp_static_staticdevice" ("id" serial NOT NULL PRIMARY KEY, "user_id" integer NOT NULL, "name" varchar(64) NOT NULL, "confirmed" boolean NOT NULL)
RELEASE SAVEPOINT "s140275211773760_x4"
SAVEPOINT "s140275211773760_x5"
CREATE TABLE "otp_static_statictoken" ("id" serial NOT NULL PRIMARY KEY, "device_id" integer NOT NULL, "token" varchar(16) NOT NULL)
RELEASE SAVEPOINT "s140275211773760_x5"
SAVEPOINT "s140275211773760_x6"
CREATE TABLE "otp_totp_totpdevice" ("id" serial NOT NULL PRIMARY KEY, "user_id" integer NOT NULL, "name" varchar(64) NOT NULL, "confirmed" boolean NOT NULL, "key" varchar(80) NOT NULL, "step" smallint NOT NULL CHECK ("step" >= 0), "t0" bigint NOT NULL, "digits" smallint NOT NULL CHECK ("digits" >= 0), "tolerance" smallint NOT NULL CHECK ("tolerance" >= 0), "drift" smallint NOT NULL, "last_t" bigint NOT NULL)
RELEASE SAVEPOINT "s140275211773760_x6"
CREATE INDEX "djkombu_queue_name_1c24e49fd475ad53_like" ON "djkombu_queue" ("name" varchar_pattern_ops)
ALTER TABLE "djkombu_message" ADD CONSTRAINT "djkombu_message_queue_id_12778caea7843dd_fk_djkombu_queue_id" FOREIGN KEY ("queue_id") REFERENCES "djkombu_queue" ("id") DEFERRABLE INITIALLY DEFERRED
CREATE INDEX "djkombu_message_46cf0e59" ON "djkombu_message" ("visible")
CREATE INDEX "djkombu_message_df2f2974" ON "djkombu_message" ("sent_at")
CREATE INDEX "djkombu_message_75249aa1" ON "djkombu_message" ("queue_id")
ALTER TABLE "otp_static_staticdevice" ADD CONSTRAINT "otp_static_staticdevice_user_id_39a61f1bd3ec970d_fk_app_user_id" FOREIGN KEY ("user_id") REFERENCES "ff_user" ("id") DEFERRABLE INITIALLY DEFERRED

So it is clear to me that my tests blow up while the test database is being setup. Specifically, the attempt to create a foreign key constraint between the otp_static_staticdevice table and my app's app_user table fails.

My immediate question is, why does django create the OTP table before my app's table? My assumption is that the OTP app is listed first in my INSTALLED_APPS. But this is not the case:

INSTALLED_APPS = [
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.admin',
    'django.contrib.humanize',
    'app',
    ...
    'django_otp',
    'django_otp.plugins.otp_static',
    'django_otp.plugins.otp_totp',
    'two_factor',
    ...
]

Next, I look at django/core/management/commands/migrate.py, trying to find out how django determines its order for migrating apps.

Plopping a pdb.set_trace() statement on line 264 (https://github.com/django/django/blob/1.8.3/django/core/management/commands/migrate.py#L264) and looking to see what app_labels contains, I get:

set(['djangosaml2', 'django_ace', 'recurly', 'staticfiles', 'distributedlock', 'app_overrides', 'messages', 'django_otp', 'kombu_transport_django', 'otp_totp', 'compressor', 'otp_static', 'humanize', 'ajax_select', 'django_extensions', 'import_export', 'raven_compat', 'crispy_forms', 'emoji'])

This is as far as I have gotten before I decided to ask for help. Does anyone know how Django might end up not creating all the project's apps in the correct order so that decency conflicts do not occur?

Answer 1

Ensure all your app directories have migrations/ folder and __init__.py file in it. If not, make a migrations/ folder and create __init__.py file inside. If you using git, add !migrations/__init__.py to the .gitignore file

Answer 2

I had a similar issue with manage.py test when I ran it in gitlab CI jobs, I was able to fix this by creating a variable called CI in my base.py settings file

base.py

"""
Set CI Variable to True when running: "manage.py test"
"""
CI = True  

and in my settings.py:

if CI:
    class DisableMigrations(object):

        def __contains__(self, item):
            return True

        def __getitem__(self, item):
            return None # For Django 1.10+

MIGRATION_MODULES = DisableMigrations()