Django manage.py test cannot load fixture properly

Question

I've written Django tests using django.test.TestCase, and I'd like to use a fixture with all of my current database data to run the tests. However, if I create the fixture as follows:

python manage.py dumpdata --indent=3 > myapp/fixtures/test_data.json

when I then run the tests using python manage.py test myapp, I get the following error:

Problem installing fixture...(traceback)
IntegrityError: Could not load auth.Permission(pk=42): duplicate key value violates unique constraint "auth_permission_content_type_id_codename_key"
DETAIL:  Key (content_type_id, codename)=(14, add_record) already exists.

I read somewhere on SO that this might be caused by a pk conflict so I then tried re-creating the fixture with:

python manage.py dumpdata --natural --indent=3 > myapp/fixtures/test_data.json

But now running the test gives me:

Problem installing fixture...(traceback)
DeserializationError: 'NoneType' object has no attribute '_meta'

I've also tried variously excluding (using the --exclude option) auth.permission and contenttypes (or both simultaneously), but then I get complaints about missing permissions (Key (permission_id)=(44) is not present in table "auth_permission".) or missing content types (DeserializationError: ContentType matching query does not exist.)

The thing is, I need permissions anyway because my tests are partly to verify that only users with specific permissions can access certain views.

I don't understand why this is happening, to be honest--my impression was that the test runner starts with a completely clean database and loads EVERYTHING from my fixture, but reading posts like this one: Django unit-testing with loading fixtures for several dependent applications problems makes it seem like maybe that's not the case.

How can I get around this? I'd much rather not have to write stuff like User.objects.create_user(.. tons of times under def setUp(self): in my tests just to have enough objects for them to run properly...

Answer 1

The issue still seems to have been present, even with --natural. However, it seems to be resolved in django1.9 with new flags: dumpdata --natural-foreign --natural-primary

see https://code.djangoproject.com/ticket/21278#comment:5

Answer 2

When you run a test the initial_data fixtures will be loaded (by syncdb).

For me dumpdata works with the --natural argument, excluding the contenttypes and then deleting some auth.permission entries manually that only these remain:

{
    "pk": 1, 
    "model": "auth.permission", 
    "fields": {
        "codename": "add_permission", 
        "name": "Can add permission", 
        "content_type": [
            "auth", 
            "permission"
        ]
    }
},
{
    "pk": 2, 
    "model": "auth.permission", 
    "fields": {
        "codename": "change_permission", 
        "name": "Can change permission", 
        "content_type": [
            "auth", 
            "permission"
        ]
    }
},
{
    "pk": 3, 
    "model": "auth.permission", 
    "fields": {
        "codename": "delete_permission", 
        "name": "Can delete permission", 
        "content_type": [
            "auth", 
            "permission"
        ]
    }
},
{
    "pk": 4, 
    "model": "auth.permission", 
    "fields": {
        "codename": "add_group", 
        "name": "Can add group", 
        "content_type": [
            "auth", 
            "group"
        ]
    }
},
{
    "pk": 5, 
    "model": "auth.permission", 
    "fields": {
        "codename": "change_group", 
        "name": "Can change group", 
        "content_type": [
            "auth", 
            "group"
        ]
    }
},
{
    "pk": 6, 
    "model": "auth.permission", 
    "fields": {
        "codename": "delete_group", 
        "name": "Can delete group", 
        "content_type": [
            "auth", 
            "group"
        ]
    }
},
{
    "pk": 7, 
    "model": "auth.permission", 
    "fields": {
        "codename": "add_user", 
        "name": "Can add user", 
        "content_type": [
            "auth", 
            "user"
        ]
    }
},
{
    "pk": 8, 
    "model": "auth.permission", 
    "fields": {
        "codename": "change_user", 
        "name": "Can change user", 
        "content_type": [
            "auth", 
            "user"
        ]
    }
},
{
    "pk": 9, 
    "model": "auth.permission", 
    "fields": {
        "codename": "delete_user", 
        "name": "Can delete user", 
        "content_type": [
            "auth", 
            "user"
        ]
    }
},
{
    "pk": 10, 
    "model": "auth.permission", 
    "fields": {
        "codename": "add_contenttype", 
        "name": "Can add content type", 
        "content_type": [
            "contenttypes", 
            "contenttype"
        ]
    }
},
{
    "pk": 11, 
    "model": "auth.permission", 
    "fields": {
        "codename": "change_contenttype", 
        "name": "Can change content type", 
        "content_type": [
            "contenttypes", 
            "contenttype"
        ]
    }
},
{
    "pk": 12, 
    "model": "auth.permission", 
    "fields": {
        "codename": "delete_contenttype", 
        "name": "Can delete content type", 
        "content_type": [
            "contenttypes", 
            "contenttype"
        ]
    }
},
{
    "pk": 13, 
    "model": "auth.permission", 
    "fields": {
        "codename": "add_session", 
        "name": "Can add session", 
        "content_type": [
            "sessions", 
            "session"
        ]
    }
},
{
    "pk": 14, 
    "model": "auth.permission", 
    "fields": {
        "codename": "change_session", 
        "name": "Can change session", 
        "content_type": [
            "sessions", 
            "session"
        ]
    }
},
{
    "pk": 15, 
    "model": "auth.permission", 
    "fields": {
        "codename": "delete_session", 
        "name": "Can delete session", 
        "content_type": [
            "sessions", 
            "session"
        ]
    }
},
{
    "pk": 16, 
    "model": "auth.permission", 
    "fields": {
        "codename": "add_site", 
        "name": "Can add site", 
        "content_type": [
            "sites", 
            "site"
        ]
    }
},
{
    "pk": 17, 
    "model": "auth.permission", 
    "fields": {
        "codename": "change_site", 
        "name": "Can change site", 
        "content_type": [
            "sites", 
            "site"
        ]
    }
},
{
    "pk": 18, 
    "model": "auth.permission", 
    "fields": {
        "codename": "delete_site", 
        "name": "Can delete site", 
        "content_type": [
            "sites", 
            "site"
        ]
    }
},

I don't understand excactly why, but it works. I'd try to compare my fixture with a dump of a fresh database directly after syncdb and decide then what to delete or edit in my fixture. Hope this helps in your case.