Why am I very-occasionally getting an ImportError using django-constance?

Question

Once every few hundred thousand requests I see one of these:

ImportError at /
cannot import name 'Config' from partially initialized module 'constance.base' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/constance/base.py)

I cannot identify any rhyme or reason. It doesn't correspond with accessing constance admin, just randomly occurs. My best guess is it's something to do with the LazyObject in constance's __init__.py, and maybe random race-conditions in restarting expired gunicorn workers or something?!

I'm using:

• Python 3.9.2
• Django 3.2
• django-constance = {extras = ["database"],version = "==2.8.*"}
• "constance" and "constance.backends.database" in INSTALLED_APPS (at top)
• CONSTANCE_BACKEND = "constance.backends.database.DatabaseBackend"
• "constance.context_processors.config" in TEMPLATES[0]["OPTIONS"]["context_processors"]

All my code does is from constance import config and access config attributes in the standard way in python code and Django templates.

For what it's worth, we've been using django-constance on this site for years, but never saw this error until we upgraded to 2.8.0 (from 2.6.0). We were using Django 3.1 when it first appeared, but has also occurred since upgrading to 3.2.

I cannot find any similar error reports on https://github.com/jazzband/django-constance/

Any ideas what could be causing this and how to resolve it?

Answer 1

This was a bug in constance that has been solved in this pull request.

The root of the problem is that prior to this pull request constance testsuite didn't run tests on Django 3.2, which is been fixed and the changes that caused your error are also outed and fixed.

That means in constance's __init__.py, now there is an if clause separating current handling of importation in Django 3.2 from the older Django versions.

Answer 2

OK, I've managed to reproduce the issue and track down the likely cause.

I managed to trigger it once using Django's runserver, on the first request handled, but in hundreds of subsequent restarts/attempts I couldn't make it recur.

I then instead ran a couple of local gunicorn worker threads with a low requests-per-worker threshold, and spammed the local port with rapid-fire requests, and sure enough the ImportError occurs every now and then.

The issue seems to be this:

• constance.__init__ uses a django.utils.functional.LazyObject which, once lazily evaluated, imports constance.base and instantiates a Config from that module.

• constance.base.Config.__init__ uses constance.utils.import_module_attr to import the backend specified in the project settings, which in turn uses importlib.import_module to import, in my case, constance.backends.database.DatabaseBackend.

• constance.backends.database.__init__ also imports config from ... (constance), thus potentially creating an import loop.

It seems there is a rare race condition in which LazyConfig._setup tries to import constance.base.Config while constance.base hasn't be fully initialised. I injected some debugging statements to demonstrate, and the buggy sequence looks like this:

[2021-09-15 09:35:10 +1000] [13504] [INFO] Booting worker with pid: 13504
BEGIN constance.__init__.LazyConfig._setup() vars(self): {'_wrapped': <object object at 0x108da8340>}
BEGIN constance.__init__.LazyConfig._setup() vars(self): {'_wrapped': <object object at 0x108da8340>}
constance.base imported
constance.base.Config.__init__: importing constance.backends.database.DatabaseBackend
ImportError at /
cannot import name 'Config' from partially initialized module 'constance.base' (most likely due to a circular import) (/usr/local/lib/python3.9/site-packages/constance/base.py) 
EXIT constance.__init__.LazyConfig._setup() vars(self): {'_wrapped': <constance.base.Config object at 0x10fa7ea30>}

It seems to only happen when _setup is called twice in a row like that, before constance.base is imported, then the two threads seem to race from that point.

I guess I'll open a GitHub issue on the django-constance repository to resolve the issue.