Django app defaults?

Question

I'm looking for a way to have application defaults and settings that are easy to use, difficult to get wrong, and have little overhead..

Currently I have it organized as follows:

myapp/defaults.py
    # application defaults
    import sys
    if sys.platform == 'win32':
         MYAPP_HOME_ROOT = os.path.dirname(os.environ['USERPROFILE'])
    else:
         MYAPP_HOME_ROOT = '/home'

in my project I have:

mysite/settings.py
    from myapp.defaults import *       # import all default from myapp
    MYAPP_HOME_ROOT = '/export/home'   # overriding myapp.defaults 

With this setup I could import and use settings in the regular django way (from django.conf import settings and settings.XXX).

update-3 (why we need this)

1. Default settings ("defaults"):
   1. An application is more convenient to use if it can be configured by overriding a set of sensible default settings.
   2. the application "has domain knowledge", so it makes sense for it to provide sensible defaults whenever possible.
   3. it isn't convenient for a user of the application to need to provide all the settings needed by every app, it should be sufficient to override a small subset and leave the rest with default values.
   4. it is very useful if defaults can react to the environment. You'll often want to do something different when DEBUG is True, but any other global setting could be useful: e.g. MYAPP_IDENTICON_DIR = os.path.join(settings.MEDIA_ROOT, 'identicons') (https://en.wikipedia.org/wiki/Identicon)
   5. project (site/global) settings must override app-defaults, i.e. a user who defined MYAPP_IDENTICON_DIR = 's3://myappbucket.example.com/identicons' in the (global) settings.py file for their site should get this value, and not the application's default.
   6. any solution that keeps close to the normal way of using settings (import .. settings; settings.FOO) is superior to a solution that needs new syntax (since new syntax will diverge and we would get new and unique ways to use settings from app to app).
   7. the zen of python probably applies here:
      • If the implementation is hard to explain, it's a bad idea.
      • If the implementation is easy to explain, it may be a good idea.

(The original post posited the two key problems below, leaving the above assumptions unstated..)

Problem #1: When running unit tests for the app there is no site however, so settings wouldn't have any of the myapp.defaults.

Problem #2: There is also a big problem if myapp.defaults needs to use anything from settings (e.g. settings.DEBUG), since you can't import settings from defaults.py (since that would be a circular import).

To solve problem #1 I created a layer of indirection:

myapp/conf.py
    from . import defaults
    from django.conf import settings

    class Conf(object):
        def __getattr__(self, attr):
            try:
                return getattr(settings, attr)
            except AttributeError:
                return getattr(defaults, attr)
    conf = Conf()  # !<-- create Conf instance

and usage:

myapp/views.py
    from .conf import conf as settings
    ...
    print settings.MYAPP_HOME_ROOT   # will print '/export/home' when used from mysite

This allows the conf.py file to work with an "empty" settings file too, and the myapp code can continue using the familiar settings.XXX.

It doesn't solve problem #2, defining application settings based on e.g. settings.DEBUG. My current solution is to add to the Conf class:

    from . import defaults
    from django.conf import settings

    class Conf(object):
        def __getattr__(self, attr):
            try:
                return getattr(settings, attr)
            except AttributeError:
                return getattr(defaults, attr)

        if settings.DEBUG:
            MYAPP_HOME_ROOT = '/Users'
        else:
            MYAPP_HOME_ROOT = '/data/media'
    conf = Conf()  # !<-- create Conf instance

but this is not satisfying since mysite can no longer override the setting, and myapp's defaults/settings are now spread over two files...

Is there an easier way to do this?

update-4: "just use the django test runner.."

The app you are testing relies on the Django framework - and you cannot get around the fact that you need to bootstrap the framework first before you can test the app. Part of the bootstrapping process is creating a default settings.py and further using the django-supplied test runners to ensure that your apps are being testing in the environment that they are likely to be run.

While that sounds like it ought to be true, it doesn't actually make much sense, e.g. there is no such thing as a default settings.py (at least not for a reusable app). When talking about integration testing it makes sense to test an app with the site/settings/apps/database(s)/cache(s)/resource-limits/etc. that it will encounter in production. For unit testing, however, we want to test just the unit of code that we're looking at - with as few external dependencies (and settings) as possible. The Django test runner(s) do, and should, mock out major parts of the framework, so it can't be said to be running in any "real" environment.

While the Django test runner(s) are great, there are a long list of issues it doesn't handle. The two biggies for us are (i) running tests sequentially is so slow that the test suite becomes unused (<5mins when running in parallel, almost an hour when running sequentially), (ii) sometimes you just need to run tests on big databases (we restore last night's backup to a test database that the tests can run against - much too big for fixtures).

The people who made nose, py.test, twill, Selenium, and any of the fuzz testing tools really do know testing well, since that is their only focus. It would be a shame to not be able to draw on their collective experience.

I am not the first to have encountered this problem, and there doesn't look like there is an easy or common solution. Here are two projects that have different solution:

Update, python-oidc-provider method:

The python-oidc-provider package (https://github.com/juanifioren/django-oidc-provider) has another creative way to solve the app-settings/defaults problem. It uses properties to define defaults in a myapp/settings.py file:

from django.conf import settings
class DefaultSettings(object):
    @property
    def MYAPP_HOME_ROOT(self):
        return ...

default_settings = DefaultSettings() 

def get(name):
    value = None
    try:
        value = getattr(default_settings, name)
        value = getattr(settings, name)
    except AttributeError:
        if value is None:
            raise Exception("Missing setting: " + name)

using a setting inside myapp becomes:

from myapp import settings
print settings.get('MYAPP_HOME_ROOT')

good: solves problem #2 (using settings when defining defaults), solves problem #1 (using default settings from tests).

bad: different syntax for accessing settings (settings.get('FOO') vs the normal settings.FOO), myapp cannot provide defaults for settings that will be used outside of myapp (the settings you get from from django.conf import settings will not contain any defaults from myapp). External code can do from myapp import settings to get regular settings and myapp defaults, but this breaks down if more than one app wants to do this...

Update2, the django-appconf package: (Note: not related to Django's AppConfig..)

With django-appconfig, app settings are created in myapp/conf.py (which needs to be loaded early, so you should probably import the conf.py file from models.py - since it is loaded early):

from django.conf import settings
from appconf import AppConf
class MyAppConf(AppConf):
    HOME_ROOT = '...'

usage:

from myapp.conf import settings
print settings.MYAPP_HOME_ROOT

AppConf will automagically add the MYAPP_ prefix, and also automagically detect if MYAPP_HOME_ROOT has been redefined/overridden in the project's settings.

pro: simple to use, solves problem #1 (accessing app-settings from tests), and problem #2 (using settings when defining defaults). As long as the conf.py file is loaded early, external code should be able to use defaults defined in myapp.

con: significantly magical. The name of the setting in conf.py is different from its usage (since appconf automatically adds the MYAPP_ prefix). Extenal/opaque dependency.

Answer 1

I've just created django-app-defaults based on all of the requirements. It's basically a generalization of the second approach highlighted in the question (class Conf(object):).

Usage:

# my_app/defaults.py

# `django.conf.settings` or any other module can be imported if needed

# required
DEFAULT_SETTINGS_MODULE = True

# define default settings below
MY_DEFAULT_SETTING = "yey"

Then anywhere within your project:

from app_defaults import settings

print(settings.MY_DEFAULT_SETTING)
# yey

# All `django.conf.settings` are also available
print(settings.DEBUG)
# True

To load default setting for a single app instead of all of the apps, just do:

# Note: when apps or modules are explicitly passed,
# the `DEFAULT_SETTINGS_MODULE` var is not required

from app_defaults import Settings

settings = Settings(apps=["my_app"])

# or

from my_app import defaults
settings = Settings(modules=[defaults])

Answer 2

I have written a django package for the management of app settings called django-pluggableappsettings.

It's a package that allows you to define sensible defaults for your settings but also adds advanced features like type checking or callable settings. It makes use of metaclasses to allow for an easy definition of the apps settings. Of course this adds an external dependency to your project.

Edit 1:

Example usage could be as follows:

Install the package using pip:

pip install django-pluggableappsettings

Create your AppSettings class in any of your project's files. E.g. in 'app_settings.py'.

app_settings.py

from django_pluggableappsettings import AppSettings, Setting, IntSetting

class MyAppSettings(AppSettings):
    MY_SETTING = Setting('DEFAULT_VALUE')
    # Checks for "MY_SETTING" in django.conf.settings and 
    # returns 'DEFAULT_VALUE' if it is not present

    ANOTHER_SETTING = IntSetting(42, aliases=['OTHER_SETTING_NAME'])
    # Checks for "ANOTHER_SETTING" or "OTHER_SETTING_NAME" in
    # django.conf.settings and returns 42 if it is not present.
    # also validates that the given value is of type int

Import your MyAppSettings in any file and access its attributes

from mypackage.app_settings import MyAppSettings
MyAppSettings.MY_SETTING
MyAppSettings.ANOTHER_SETTING

Note, that the settings are only initialized on first access, so if you never access a setting, its presence in django.conf.settings is never checked.