Django unable to load test fixtures, IntegrityError

Question

I'm using Saleor/Satchless to power and e-commerce site (inherited the project). I couldn't find a Saleor mailing list so posting here instead.

I dumped the DB to create some test fixtures.

./manage.py dumpdata -e contenttypes -e sessions -e south -e > payments_data.json

When I run the tests and django tries to load the fixtures it barfs this up. I'm using Postgres as the DB, and although not terribly familiar, it seems like there might be a problem with the order in which data is being loaded

Any ideas how to get around it?

======================================================================
ERROR: test__hometryon_extra_pair_sizes (payment.tests.InterstitialTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/test/testcases.py", line 259, in __call__
self._pre_setup()
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/test/testcases.py", line 479, in _pre_setup
self._fixture_setup()
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/test/testcases.py", line 849, in _fixture_setup
'skip_validation': True,
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/core/management/__init__.py", line 161, in call_command
return klass.execute(*args, **defaults)
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/core/management/base.py", line 255, in execute
output = self.handle(*args, **options)
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 224, in handle
connection.check_constraints(table_names=table_names)
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 132, in check_constraints
self.cursor().execute('SET CONSTRAINTS ALL IMMEDIATE')
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 56, in execute
six.reraise(utils.IntegrityError, utils.IntegrityError(*tuple(e.args)), sys.exc_info()[2])
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 54, in execute
return self.cursor.execute(query, args)
IntegrityError: Problem installing fixtures: insert or update on table "product_ship" violates foreign key constraint "product_ship_product_ptr_id_fkey"
DETAIL:  Key (product_ptr_id)=(1) is not present in table "product_product".

Update: Started playing around with django-fixture-magic I discovered that product.product is not actually being dumped even when I do a $ python manage.py dumpdata --indent=4 --all -e contenttypes -e sessions -e south > fulldb.json

Update 2: After including contenttypes in the dumpdata, product.product was still missing in the actual data. I had to manually perform a dump_object and merge_fixture for product.product and order_deliverygroup. When loading the fixture, both gave similar errors to the following:

IntegrityError: Problem installing fixtures: insert or update on table "order_shippeddeliverygroup" violates foreign key constraint "order_shippeddeliverygroup_deliverygroup_ptr_id_fkey" DETAIL:  Key (deliverygroup_ptr_id)=(1) is not present in table "order_deliverygroup".

I am now left with this problem around Contenttypes

IntegrityError: Problem installing fixture '/Users/andres/Documents/projects/rpmwest/rpmwest/payment/fixtures/payments_data.json': Could not load contenttypes.ContentType(pk=5): duplicate key value violates unique constraint "django_content_type_app_label_model_key"DETAIL:  Key (app_label, model)=(product, digitalship) already exists.

Looking at the data in the fixture, sure enough it's there, but there is only one of them. Is it clashing with the DB creating contenttypes when it initially syncs the models?

If I leave out the contenttypes I get the following error:

Traceback (most recent call last):
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/test/testcases.py", line 259, in __call__
self._pre_setup()
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/test/testcases.py", line 479, in _pre_setup
self._fixture_setup()
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/test/testcases.py", line 849, in _fixture_setup
'skip_validation': True,
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/core/management/__init__.py", line 161, in call_command
return klass.execute(*args, **defaults)
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/core/management/base.py", line 255, in execute
output = self.handle(*args, **options)
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 224, in handle
connection.check_constraints(table_names=table_names)
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 132, in check_constraints
self.cursor().execute('SET CONSTRAINTS ALL IMMEDIATE')
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 56, in execute
six.reraise(utils.IntegrityError, utils.IntegrityError(*tuple(e.args)), sys.exc_info()[2])
  File "/Users/andres/.virtualenvs/rpmwest/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 54, in execute
return self.cursor.execute(query, args)
IntegrityError: Problem installing fixtures: insert or update on table "django_admin_log" violates foreign key constraint "django_admin_log_content_type_id_fkey" DETAIL:  Key (content_type_id)=(22) is not present in table "django_content_type".

Answer 1

Seems that django dumpdata dumped fixtures in wrong order. Look in json file to check whether product with id: 1is present there. If, as I suppose, this is true, use some more sophisticated tools to dump data, for example django-fixture-magic

Alternatively, you might want to delete all integrity constraints by means of db engine just before upload and try to recreate them right after, but this is somewhat risky if some integrity errors will be present.

For PostgreSQL, consult this thread to know how to get your tables definitions. In MySQL it would be something like follows:

$ mysqldump --no-data -utest django auth_user_user_permissions
CREATE TABLE `auth_user_user_permissions` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `user_id` int(11) NOT NULL,
  `permission_id` int(11) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `user_id` (`user_id`,`permission_id`),
  KEY `auth_user_user_permissions_403f60f` (`user_id`),
  KEY `auth_user_user_permissions_1e014c8f` (`permission_id`),
  CONSTRAINT `user_id_refs_id_dfbab7d` FOREIGN KEY (`user_id`) REFERENCES `auth_user` (`id`),
  CONSTRAINT `permission_id_refs_id_67e79cb` FOREIGN KEY (`permission_id`) REFERENCES `auth_permission` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Following part should work for oracle, postgre and mysql

> alter table `auth_user_user_permissions` drop foreign key `user_id_refs_id_dfbab7d`;
Query OK, 0 rows affected (0.97 sec)
Records: 0  Duplicates: 0  Warnings: 0

> alter table `auth_user_user_permissions` add CONSTRAINT `user_id_refs_id_dfbab7d` FOREIGN KEY (`user_id`) references `auth_user` (`id`);
Query OK, 0 rows affected (0.95 sec)
Records: 0  Duplicates: 0  Warnings: 0