Django sqlite3 timeout has no effect

Question

I have a simple integration test in Django that spawns a single Celery worker to run a job, which writes a record to the database. The Django thread also writes a record to the database. Because it's a test, I use the default in-memory sqlite3 database. There are no transactions being used.

I often get this error:

django.db.utils.OperationalError: database table is locked

which according to the Django docs is due to one connection timing out while waiting for another to finish. It's "more concurrency than sqlite can handle in default configuration". This seems strange given that it's two records in two threads. Nevertheless, the same docs say to increase the timeout option to force connections to wait longer. Ok, I change my database settings to this:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
        'OPTIONS': {'timeout': 10000000},
    }
}

This has no effect. The error still appears and it clearly has not waited 1e7 seconds or 1e7 milliseconds or 1e7 microseconds before doing so. Is there an additional setting I'm missing?

I have tried both Python 3.5 and Python 3.6 and both Django 1.11 and Django 2.0.

Answer 1

I had the same issue and my experiments gave me the following:

1. I've figured out that Django uses in-memory SQLite DB in the test mode until you explicitly change this. That explains why I only see that problem in my unit tests. To force Django to use SQLite DB in the file set DATABASES->TEST->NAME explicitly in your settings.py. For example like this:

   DATABASES = {
       'default': {
           ...
           'TEST': {
               'NAME': 'testdb.sqlite3',
           },
       },
   }
   

2. Setting timeout value larger than 2147483.647 (looks familiar, right? :-) ) disables timeout (or sets it to negligibly small value).

3. As far as I understand, the root of the problem is that when SQLite uses the shared cache the timeout value is not respected at all.