Deploying Django app on Heroku: Can I manually set environment variables in the .env file? Do I need to install tools like autoenv, heroku-config...?

Question

My goal:

I intend to follow "The Twelve-Factor App" methodology for building my Django app on Heroku.

Introduction:

I'm following the "Getting Started with Django on Heroku" quick start guide. At the moment I have the following directory structure:

~/Projects/
    hellodjango_rep/
        .env (empty)
        .git
        .gitignore
        Procfile
        requirements.txt
        hellodjango/
            manage.py
            hellodjango/
                __init__.py
                settings/
                urls.py
                wsgi.py

I installed django-toolbelt, created my simple Django application, started the process in my Procfile... Everything seemed to be working fine, but the problems started when I configured the application for the Heroku environment and added:

import dj_database_url
DATABASES['default'] =  dj_database_url.config()

to the bottom of my settings.py file.

I pushed my application’s repository to Heroku, visited the app in my browser with $ heroku open successfully, but locally: dj_database_url.config() returned an empty dictionary.

Locally:

OS X 10.8.4
pip==1.4.1
virtualenv==1.10.1
virtualenvwrapper==4.1.1
wsgiref==0.1.2
Postgres.app running on Port 5432

Environment variables:

mac-pol:hellodjango_rep oubiga$ python
>>> import os
>>> os.environ

{
  'PROJECT_HOME': '/Users/oubiga/Projects'... 
  'PATH': '/usr/local/heroku/bin:/usr/local/share/python:/usr/local/bin:/Applications/Postgres.app/Contents/MacOS/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin'... 
  'HOME': '/Users/oubiga'... 
  'WORKON_HOME': '/Users/oubiga/Envs'... 
  'VIRTUALENVWRAPPER_HOOK_DIR': '/Users/oubiga/Envs'... 
  'PWD': '/Users/oubiga/Projects/hellodjango_rep'
}

hellodjango_venv:

Django==1.5.2
dj-database-url==0.2.2
dj-static==0.0.5
django-toolbelt==0.0.1
gunicorn==18.0
psycopg2==2.5.1
static==0.4

This is what I have in my wsgi.py file:

import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "hellodjango.hellodjango.settings")
from django.core.wsgi import get_wsgi_application
from dj_static import Cling
application = Cling(get_wsgi_application())

This is what I have in my manage.py file:

import os
import sys
if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "hellodjango.settings")
    from django.core.management import execute_from_command_line
    execute_from_command_line(sys.argv)

This is what I have in my Procfile:

web: gunicorn hellodjango.hellodjango.wsgi

Environment variables:

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ python hellodjango/manage.py shell
>>> import os
>>> os.environ 


{
  'PROJECT_HOME': '/Users/oubiga/Projects'... 
  'PATH': '/Users/oubiga/Envs/hellodjango_venv/bin:/usr/local/heroku/bin:/usr/local/share/python:/usr/local/bin:/Applications/Postgres.app/Contents/MacOS/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/git/bin', 
  'HOME': '/Users/oubiga'... 
  'WORKON_HOME': '/Users/oubiga/Envs'... 
  'VIRTUAL_ENV': '/Users/oubiga/Envs/hellodjango_venv'...
  'VIRTUALENVWRAPPER_HOOK_DIR': '/Users/oubiga/Envs'... 
  'PWD': '/Users/oubiga/Projects/hellodjango_rep'... 
  'DJANGO_SETTINGS_MODULE': 'hellodjango.settings'
}

On Heroku:

Environment variables:

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ heroku run python hellodjango/manage.py shell
>>> import os
>>> os.environ

{
  'DATABASE_URL': 'postgres://dbuser:dbpassword@ec2-184-73-162-34.compute-1.amazonaws.com:5432/dbname', 
  'HEROKU_POSTGRESQL_ORANGE_URL': 'postgres://dbuser:dbpassword@ec2-184-73-162-34.compute-1.amazonaws.com:5432/dbname', 
  'LIBRARY_PATH': '/app/.heroku/vendor/lib', 'PWD': '/app'... 
  'DJANGO_SETTINGS_MODULE': 'hellodjango.settings', 
  'PYTHONHOME': '/app/.heroku/python'... 
  'PYTHONPATH': '/app/'... 
  'DYNO': 'run.9068', 
  'LD_LIBRARY_PATH': '/app/.heroku/vendor/lib'... 
  'HOME': '/app', '_': '/app/.heroku/python/bin/python', 
  'PATH': '/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin'...
}

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ heroku config
=== damp-dusk-5382 Config Vars
DATABASE_URL: postgres://dbuser:dbpassword@ec2-184-73-162-34.compute-1.amazonaws.com:5432/dbname
HEROKU_POSTGRESQL_ORANGE_URL: postgres://dbuser:dbpassword@ec2-184-73-162-34.compute-1.amazonaws.com:5432/dbname

Research:

Store config in the environment: from The Twelve-Factor App

@adamwiggins wrote:

The twelve-factor app stores config in environment variables... Env vars are easy to change between deploys without changing any code; unlike config files.

dj_database_url.config() is returning an empty object: from Heroku Forums

@chrisantonick replied:

... dj_database_url.config() gets the Postgres credentials from the Heroku environment variables. But, on your local machine, those variables aren't there. You have to put them in your /venv/bin/activate shell script... put the variables in there. something like
DATABASE_URL = "xxx"
export DATABASE_URL
For each thing it needs. Then... "deactivate"... and ..."activate" again to restart it.

Getting Started with Django and Heroku instructions raised ImproperlyConfigured error: from Heroku Forums

@jwpe replied:

... dj-database-url is a great utility, as it allows you to use exactly the same settings.py code in your development and production environments, as recommended in the "12 factor app principles"... what dj_database_url.config() is doing is looking for the DATABASE_URL environment variable, and then parsing it into Django's preferred format... if you haven't manually created and promoted a postgres DB on Heroku, DATABASE_URL will not be present and the ImproperlyConfigured error will be raised. Setting the default for dj_database_url.config() as your local DB URL is one way to make sure that your application will work in a development environment. However, it is not necessarily the only way. Perhaps a better alternative is to manually set DATABASE_URL in your local .env file. Then, when running your app locally using Foreman, it will be loaded as an environment variable and dj_database_url will find it. So your .env would contain:

DATABASE_URL=postgres://user:pass@localhost/dbname

Meaning that in settings.py you would only need to have:

DATABASES['default']= dj_database_url.config()

...The advantage of using a local environment variable instead of a single, hard-coded default is that your code will run in any environment where the DATABASE_URL is set. If you change the name of your local DB, or want to run your code on a different dev machine, you only need to update your .env file instead of tinkering with settings.py.

How to manage production/staging/dev Django settings?: from Heroku Forums

@rdegges replied:

... trying to get your application to behave in a way such that:

When you're running the app on your laptop, it uses your local Postgres server.

When you're running the app on your staging Heroku app, it uses the Postgres server addon.

When you're running the app on your production Heroku app, it uses the Postgres server addon.

The best way to accomplish this is by using environment variables!… Environment variables are the most elegant (and scalable) way to handle application configuration between different environments… Instead of having many settings files, define a single file: settings.py, and have it make use of environment variables to pull service information and credentials... On Heroku, you can set environment variables manually by running:
$ heroku config:set SOME_VARIABLE=some_value
... there's always Kenneth Reitz's great autoenv tool. This lets you define a simple .env file in your project directory… And each time you enter your project directory, those environment variables will be automatically set so that you don't have to do anything special! Just run your project and everything will work as expected: python manage.py runserver

As a First Attempt:

I manually set DATABASE_URL in my .env file: DATABASE_URL=postgres://dbuser:dbpassword@ec2-184-73-162-34.compute-1.amazonaws.com:5432/dbname

But when I run $ foreman start command:

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ foreman start
17:25:39 web.1  | started with pid 319
17:25:39 web.1  | 2013-09-11 17:25:39 [319] [INFO] Starting gunicorn 18.0
17:25:39 web.1  | 2013-09-11 17:25:39 [319] [INFO] Listening at: http://0.0.0.0:5000 (319)
17:25:39 web.1  | 2013-09-11 17:25:39 [319] [INFO] Using worker: sync
17:25:39 web.1  | 2013-09-11 17:25:39 [322] [INFO] Booting worker with pid: 322

and tried to open my app in the browser http://0.0.0.0:5000:

17:26:59 web.1  | 2013-09-11 10:26:59 [322] [ERROR] Error handling request
17:26:59 web.1  | Traceback (most recent call last):
17:26:59 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 131, in handle_request
17:26:59 web.1  |     respiter = self.wsgi(environ, resp.start_response)
17:26:59 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/dj_static.py", line 59, in __call__
17:26:59 web.1  |     return self.application(environ, start_response)
17:26:59 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 236, in __call__
17:26:59 web.1  |     self.load_middleware()
17:26:59 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 53, in load_middleware
17:26:59 web.1  |     raise exceptions.ImproperlyConfigured('Error importing middleware %s: "%s"' % (mw_module, e))
17:26:59 web.1  | ImproperlyConfigured: Error importing middleware django.contrib.auth.middleware: "dlopen(/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/psycopg2/_psycopg.so, 2): Library not loaded: @loader_path/../lib/libssl.1.0.0.dylib
17:26:59 web.1  |   Referenced from: /Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/psycopg2/_psycopg.so
17:26:59 web.1  |   Reason: image not found"

However, dj_database_url.config() returned:

{
  'ENGINE': 'django.db.backends.postgresql_psycopg2', 
  'NAME': 'dbname', 
  'HOST': 'ec2-23-21-196-147.compute-1.amazonaws.com', 
  'USER': 'dbuser', 
  'PASSWORD': 'dbpassword', 
  'PORT': 5432
}

As a Second Attempt:

I manually set DATABASE_URL in my .env file changing the host. I replaced "ec2-184-73-162-34.compute-1.amazonaws.com:5432" by "localhost:5000". $ deactivate and then $ workon hellodjango_venv again.

DATABASE_URL=postgres://dbuser:dbpassword@localhost:5000/dbname

But, when I run $ foreman start command:

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ foreman start
17:38:41 web.1  | started with pid 687
17:38:41 web.1  | 2013-09-11 17:38:41 [687] [INFO] Starting gunicorn 18.0
17:38:41 web.1  | 2013-09-11 17:38:41 [687] [INFO] Listening at: http://0.0.0.0:5000 (687)
17:38:41 web.1  | 2013-09-11 17:38:41 [687] [INFO] Using worker: sync
17:38:41 web.1  | 2013-09-11 17:38:41 [690] [INFO] Booting worker with pid: 690

and tried to open my app in the browser http://0.0.0.0:5000:

17:38:46 web.1  | 2013-09-11 10:38:46 [690] [ERROR] Error handling request
17:38:46 web.1  | Traceback (most recent call last):
17:38:46 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 131, in handle_request
17:38:46 web.1  |     respiter = self.wsgi(environ, resp.start_response)
17:38:46 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/dj_static.py", line 59, in __call__
17:38:46 web.1  |     return self.application(environ, start_response)
17:38:46 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 236, in __call__
17:38:46 web.1  |     self.load_middleware()
17:38:46 web.1  |   File "/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 53, in load_middleware
17:38:46 web.1  |     raise exceptions.ImproperlyConfigured('Error importing middleware %s: "%s"' % (mw_module, e))
17:38:46 web.1  | ImproperlyConfigured: Error importing middleware django.contrib.auth.middleware: "dlopen(/Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/psycopg2/_psycopg.so, 2): Library not loaded: @loader_path/../lib/libssl.1.0.0.dylib
17:38:46 web.1  |   Referenced from: /Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/psycopg2/_psycopg.so
17:38:46 web.1  |   Reason: image not found"

This time, dj_database_url.config() returned:

{
  'ENGINE': 'django.db.backends.postgresql_psycopg2', 
  'NAME': 'dbname', 
  'HOST': 'localhost', 
  'USER': 'dbuser', 
  'PASSWORD': 'dbpassword', 
  'PORT': 5000
}

As a Third Attempt:

I installed autoenv mac-pol:~ oubiga$ pip install autoenv
From this Cookbook Kenneth Reitz wrote, I put:

use_env() {
  typeset venv
  venv="$1"
  if [[ "${VIRTUAL_ENV:t}" != "$venv" ]]; then
    if workon | grep -q "$venv"; then
      workon "$venv"
    else
      echo -n "Create virtualenv $venv now? (Yn) "
      read answer
      if [[ "$answer" == "Y" ]]; then
        mkvirtualenv "$venv"
      fi
    fi
  fi
}

in my .bashrc file.

I run $ foreman start command:

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ foreman start
18:11:57 web.1  | started with pid 1104
18:11:57 web.1  | 2013-09-11 18:11:57 [1104] [INFO] Starting gunicorn 18.0
18:11:57 web.1  | 2013-09-11 18:11:57 [1104] [INFO] Listening at: http://0.0.0.0:5000 (1104)
18:11:57 web.1  | 2013-09-11 18:11:57 [1104] [INFO] Using worker: sync
18:11:57 web.1  | 2013-09-11 18:11:57 [1107] [INFO] Booting worker with pid: 1107

and tried to open my app in the browser http://0.0.0.0:5000: It worked!

^CSIGINT received
18:12:06 system | sending SIGTERM to all processes
18:12:06 web.1  | 2013-09-11 11:12:06 [1107] [INFO] Worker exiting (pid: 1107)
SIGTERM received
18:12:06 web.1  | 2013-09-11 18:12:06 [1104] [INFO] Handling signal: int
18:12:06 web.1  | 2013-09-11 18:12:06 [1104] [INFO] Shutting down: Master
18:12:06 web.1  | exited with code 0

But, dj_database_url.config() again returns an empty dictionary.

As a Final Attempt:

I was curious about the python manage.py runserver command and I checked it out.

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ foreman run python hellodjango/manage.py runserver
Validating models...

0 errors found
September 11, 2013 - 18:42:37
Django version 1.5.2, using settings 'hellodjango.settings'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

and tried to open my app in the browser http://127.0.0.1:8000/: It didn't work!

An ImportError: No module named hellodjango.urls was raised.

I replaced ROOT_URLCONF = 'hellodjango.hellodjango.urls' in my settings.py file by ROOT_URLCONF = 'hellodjango.urls' and it finally worked.

As expected, dj_database_url.config() returned an empty dictionary.

So:

Now, I feel a little overwhelmed. I'm afraid I'm misunderstanding some core concept here.

What's the point to use gunicorn instead of the Django development server?
Why does dj_database_url.config() sometimes return a fully populated dictionary, and sometimes an empty one?
Can I manually set environment variables in the .env file? Do I need to install tools like autoenv, heroku-config...?

Thank you in advance.

Song Keang · Accepted Answer

I got stuck with the postgres as well, here's what I did in the settings.py to add local settings:

DATABASES = {
'default': dj_database_url.config(default='postgres://<user>:<password>@<host>/<dbname>')
}

Of course you have to have created the database following postgres steps. Solution was from https://discussion.heroku.com/t/dj-database-url-config-is-returning-an-empty-object/55/9

Deploying Django app on Heroku: Can I manually set environment variables in the .env file? Do I need to install tools like autoenv, heroku-config...?

Tags:

environment-variables

django

heroku

gunicorn

12factor

My goal:

Introduction:

Locally:

hellodjango_venv:

On Heroku:

Research:

As a First Attempt:

As a Second Attempt:

As a Third Attempt:

As a Final Attempt:

So:

oubiga

1 Answers

Song Keang

Recent Activity

Donate For Us

Deploying Django app on Heroku: Can I manually set environment variables in the .env file? Do I need to install tools like autoenv, heroku-config...?

Tags:

environment-variables

django

heroku

gunicorn

12factor

My goal:

Introduction:

Locally:

hellodjango_venv:

On Heroku:

Research:

As a First Attempt:

As a Second Attempt:

As a Third Attempt:

As a Final Attempt:

So:

oubiga

1 Answers

Song Keang

Related questions

Recent Activity

Donate For Us