Elegantly handle site-specific settings/configuration in svn/hg/git/etc?

Question

I've been looking for a better way to deal with site-specific settings (in this case, the django settings.py file).

The settings.py structure and fields are fairly consistent, but the values differ between the developer's boxes, the integration, QA, testing, and production environments.

What's an elegant way to have the settings source controlled while still allowing changes between different boxes?

I'm also concerned about having sensitive data (eg. database passwords) in source control, but I do want automated deployments.

Examples of what we've used:

• settings.py sets the common values then loads a secondary settings file based on the hostname or the username .

• injecting values into the settings.py file using a deployment script. But this simply shifts the problem to managing the deployment scripts instead of the settings.py script.

Anyone have a particularly elegant approach?

Answer 1

Create a main settings.py file, which should include this:

# Pull in hostname-based changes.
import socket
HOSTNAME = socket.gethostname().lower().split('.')[0].replace('-','')

try:
    exec "from myproject.settings.host_%s import *" % HOSTNAME
except ImportError:
    pass

# Pull in the local changes.
try:
    from myproject.settings.local import *
except ImportError:
    pass

Now you create a new settings file for each host name you care about. But these are really small. Each of your production server's file just contains:

from myproject.settings.production import *

and your staging servers have:

from myproject.settings.staging import *

Now you can create a production.py file with production overrides for settings, a staging.py, and so on. You can make new files for each role a server plays.

Finally, you can create a local.py file on any machine (including developers' machines) with local overrides, and mark this file as ignored by source control, so that changes don't get checked in.

We've used this structure for years, it works really well.

Answer 2

+1 for Ned's answer, but want to mention a slight variation.

I see Django settings as falling into 2 areas: project and instance.

Project settings are common to all the instances (dev, testing, production, multiple sites in production) and instance settings are local to just that specific server instance.

Project settings are things like INSTALLED_APPS (although local settings may also include this to add things like django debug toolbar for developers), MIDDLEWARE_CLASSES, and TEMPLATE_LOADERS. Instance settings are things like the database settings, MEDIA_URL settings, etc.

Project settings go in settings.py and instance settings in local_settings.py, which is imported into settings.py. local_settings.py is listed in .gitignore and project settings are stored in git.

I've tried several other variations on this but ended here because it's just so much simpler.

The only time I don't like this setup is for multiple sites (using Django sites framework), which end up proliferating into things like sitename_settings.py which imports sitename_local_settings.py etc.

Finally, I do keep a local_settings_template.py in git, to use as a starting point for new instances and for devs to track changes they might need to their own local settings.

Answer 3

Lets separate those two distinct problems: 1) managing site-specific settings and 2) managing secrets.

1) Site-specific settings

Version everything (except secrets), even developer-specific settings.

With Django and a lot of other softwares, the configuration file is a piece of executable code, which makes it easy to load common configuration settings and override whatever needs to be overridden. This way you can stay DRY.

# settings_prod.py
from settings_base import *
... # override whatever needs to be overridden for production environment

So now you have settings_base.py, settings_prod.py, settings_dev.py, settings_developper_john.py, etc. How do you tell Django which one to use?

Deploying the appropriate settings file to the server is a task for the deployment script, I believe. The deployment script would know that you're deploying to host prod17 which is a production server, so it would generate on the fly a settings.py file that would look like this:

# settings.py (generated by deployment script)
from settings_prod import *

Another solution is to have that logic in a generic settings.py: it could read an environment variable or get the host name (or apply any other logic) and load the appropriate settings module:

# settings.py
import os
if os.environ["MY_APP_ENV"] == "prod":
    from settings_prod import *
elif ...

My favorite solution for Django settings is described here.

For any other software that is not as flexible with it's configuration file, the best option is probably to have the deployment script generate the configuration file, possibly using templates (tools like Chef or Puppet make this easy). This allows you to stay DRY: for example, say a software requires a flat config.ini file, then the deployment script could read a common.ini and a production.ini file, mix them together appropriately and produce a config.ini ready to be deployed to production.

Managing secrets

First of all, do not store your passwords in a version control system. :-)

One solution for managing secrets is to have the deployment script transfert the secrets. For example, bob is responsible for the deployment of web applications, he knows the password to the database, so when he launches the deployment script, he is prompted for the database password, and the script transfers it to the server. Or the deployment script simply reads the password in a file on bob's computer and transfers it. This is probably the most common solution. It's fine in most cases.

              secrets
deployer ================> server

If you need to automate the creation of VMs and you do not want the automated-deployer to know any secret, then you could include the secrets in the VM-image. Of course someone must include the secrets in the VM image in the first place.

                  VM image including secrets
human deployer -------------------------------+
                                              |
                                              |
                    image_name                v
automated deployer ==============> Cloud Service ========> VM including secrets

The problem with this solution is that you need to generate a new VM image every time any secret changes. If you want to avoid that, then you might want a "secret-server": a server to manage every other server's secrets. Then the only secret you need to include in the VM image is the bootstrap secret needed to connect to the "secret-server".

step 1:

               VM image including bootstrap secret
human deployer -----------------------------------+
                                                  |
                                                  |
                    image_name                    v
automated deployer ==================> Cloud Service ========> VM including secrets


step 2:

    bootstrap secret
   ==================>
VM                     Secret Server
   <==================
         secrets

For example, the secret server could be a Chef server, the secrets could be store in encrypted data bags, and the bootstrap secret would be the key to decrypt these bags.