How do I run Django tests on a copy of my production database?

Question

I've written a series of tests for my Django app, and would like to run them on a copy of my production database.

As far as I can tell, the best way to do this is using fixture loading like so:

• Run manage.py dumpdata -o app.dump
• Move the resulting app.dump file to a fixtures directory in the [app name] folder
• Specify a 'fixtures' class attribute on my django.test.TestCase subclass

However, this approach is cumbersome. I have multiple apps, and running manage.py dumpdata for each of them and manually moving around fixtures files every time I want to test my app is a pain.

Is there an easier way to automatically generate a copy of my entire production database and test my Django apps against it?

Answer 1

Generally, testing against the live DB or a copy of the live DB is discouraged. Why? Because tests need to be predictable. When you make a copy of the live db, the input becomes unpredictable. The second problem is that you cannot obviously test on the live site, so you need to clone the data. That's slow for anything more than few MB in size.

Even if the DB is small, dumpdata followed by loaddata isn't the way. That's because dumpdata by default exports in a JSON format which has a large generating overhead, not to mention making the data file very bulky. Importing using loaddata is even slower.

The only realistic way to make a clone is using the database engines built in export/import mechanism. In the case of sqlite that's just copying the db file. For mysql it's SELECT INTO OUTFILE followed by LOAD DATA INFILE. And for postgresql it's COPY TO followed by COPY FROM and so on.

All of these export/import commands can be executed using the lowlevel connection object available in django and thus can be used to load fixtures.