How to run Django's test database only in memory?

Question

If you set your database engine to sqlite3 when you run your tests, Django will use a in-memory database.

I'm using code like this in my settings.py to set the engine to sqlite when running my tests:

if 'test' in sys.argv:
    DATABASE_ENGINE = 'sqlite3'

Or in Django 1.2:

if 'test' in sys.argv:
    DATABASES['default'] = {'ENGINE': 'sqlite3'}

And finally in Django 1.3 and 1.4:

if 'test' in sys.argv:
    DATABASES['default'] = {'ENGINE': 'django.db.backends.sqlite3'}

(The full path to the backend isn't strictly necessary with Django 1.3, but makes the setting forward compatible.)

You can also add the following line, in case you are having problems with South migrations:

    SOUTH_TESTS_MIGRATE = False

I usually create a separate settings file for tests and use it in test command e.g.

python manage.py test --settings=mysite.test_settings myapp

It has two benefits:

1. You don't have to check for test or any such magic word in sys.argv, test_settings.py can simply be

   from settings import *
   
   # make tests faster
   SOUTH_TESTS_MIGRATE = False
   DATABASES['default'] = {'ENGINE': 'django.db.backends.sqlite3'}
   

   Or you can further tweak it for your needs, cleanly separating test settings from production settings.

2. Another benefit is that you can run test with production database engine instead of sqlite3 avoiding subtle bugs, so while developing use

   python manage.py test --settings=mysite.test_settings myapp
   

   and before committing code run once

   python manage.py test myapp
   

   just to be sure that all test are really passing.

MySQL supports a storage engine called "MEMORY", which you can configure in your database config (settings.py) as such:

    'USER': 'root',                      # Not used with sqlite3.
    'PASSWORD': '',                  # Not used with sqlite3.
    'OPTIONS': {
        "init_command": "SET storage_engine=MEMORY",
    }

Note that the MEMORY storage engine doesn't support blob / text columns, so if you're using django.db.models.TextField this won't work for you.

I can't answer your main question, but there are a couple of things that you can do to speed things up.

Firstly, make sure that your MySQL database is set up to use InnoDB. Then it can use transactions to rollback the state of the db before each test, which in my experience has led to a massive speed-up. You can pass a database init command in your settings.py (Django 1.2 syntax):

DATABASES = {
    'default': {
            'ENGINE':'django.db.backends.mysql',
            'HOST':'localhost',
            'NAME':'mydb',
            'USER':'whoever',
            'PASSWORD':'whatever',
            'OPTIONS':{"init_command": "SET storage_engine=INNODB" } 
        }
    }

Secondly, you don't need to run the South migrations each time. Set SOUTH_TESTS_MIGRATE = False in your settings.py and the database will be created with plain syncdb, which will be much quicker than running through all the historic migrations.

You can do double tweaking:

• use transactional tables: initial fixtures state will be set using database rollback after every TestCase.
• put your database data dir on ramdisk: you will gain much as far as database creation is concerned and also running test will be faster.

I'm using both tricks and I'm quite happy.

How to set up it for MySQL on Ubuntu:

$ sudo service mysql stop
$ sudo cp -pRL /var/lib/mysql /dev/shm/mysql

$ vim /etc/mysql/my.cnf
# datadir = /dev/shm/mysql
$ sudo service mysql start

Beware, it's just for testing, after reboot your database from memory is lost!

Another approach: have another instance of MySQL running in a tempfs that uses a RAM Disk. Instructions in this blog post: Speeding up MySQL for testing in Django.

Advantages:

• You use the exactly same database that your production server uses
• no need to change your default mysql configuration