How to run Django tests on Heroku

Question

I have a app that is deployed to Heroku, and I'd like to be able to run the test suite post-deployment on the target environment. I am using the Heroku Postgres add-on, which means that I have access to a single database only. I have no rights to create new databases, which in turn means that the standard Django test command fails, as it can't create the test_* database.

$ heroku run python manage.py test Running `python manage.py test` attached to terminal... up, run.9362 Creating test database for alias 'default'... Got an error creating the test database: permission denied to create database 

Is there any way around this?

Answer 1

Turns out I was in the wrong. I was not testing what I thought was being tested... Since Heroku's Routing Mesh was sending requests to different servers, the LiveServerTestCase was starting a web server on one machine and Selenium was connecting to other machines altogether.

By updating the Heroku Procfile to:

web: python src/manage.py test --liveserver=0.0.0.0:$PORT

overriding the DATABASES setting to point to the test database, and customizing the test suite runner linked to below (the same idea still holds: override setup_databases so that it only drops/re-creates tables, not the entire database), I was able to run remote tests. But this is even more hacky/painful/inelegant. Still looking for something better! Sorry about the confusion.

---

(updated answer below)

Here are the steps that worked for me:

• Create an additional, free Postgres database using the Heroku toolbelt

heroku addons:add heroku-postgresql:dev

• Use the HerokuTestSuiteRunner class which you'll find here.

This custom test runner requires that you define a TEST_DATABASES setting which follows the typical DATABASES format. For instance:

TEST_DATABASES = {     'default': dj_database_url.config(env='TEST_DATABASE_URL') } 

Then, have the TEST_RUNNER setting be a Python path to wherever HerokuTestSuiteRunner can be found.

You should now be able to run Django tests on Heroku using the given database. This is very much a quick hack... Let me know how it could be improved / made less hackish. Enjoy!

---

(original answer below)

A few relevant solutions have been discussed here. As you can read in the Django docs, "[w]hen using the SQLite database engine, the tests will by default use an in-memory database".

Although this doesn't thoroughly test the database engine you're using on Heroku (I'm still on the lookout for a solution that does that), setting the database engine to SQLite will at least allow you to run your tests.

See the above-linked StackOverflow question for some pointers. There are at least two ways out: testing if 'test' in sys.argv before forcing SQLite as the database engine, or having a dedicated settings file used in testing, which you can then pass to django manage.py test using the --settings option.

Answer 2

Starting with version 1.8, Django now has an option called keepdb, which allows for the same database to be reused during tests.

The --keepdb option can be used to preserve the test database between test runs.

This has the advantage of skipping both the create and destroy actions which can greatly decrease the time to run tests, especially those in a large test suite.

If the test database does not exist, it will be created on the first run and then preserved for each subsequent run.

Any unapplied migrations will also be applied to the test database before running the test suite.

Since it also allows for the test database to exist prior to running tests, you can simply add a new Postgres Heroku instance to your dyno and configure the tests to use that particular database.

Bonus : you can also use the failfast option, which exits as soon as your first test crashes, so that you don't have to wait for all tests to complete.

However, if you are deploying things to Heroku and you are using Heroku Pipelines, an even better option is available : Heroku CI.