Is it possible to have a Procfile and a manage.py file in a different folder level?

Question

Introduction:

• I am following the Getting Started with Django on Heroku quick start guide.

• I intend to apply the Two Scoops of Django book philosophy about working with virtualenvs and projects. Audrey Roy and Daniel Greenfeld (authors) like to put all the environments in one directory and all the projects in another.

• I also intend to apply the Two Scoops of Django book philosophy about placing what is generated by the django-admin.py startproject management command inside another directory which serves as the git repository root (a.k.a. three-tiered approach).

The layout at the highest level:

repository_root/
    django_project_root/
        configuration_root/

Locally:

OS X 10.8.4
pip==1.4.1
virtualenv==1.10.1
virtualenvwrapper==4.1.1
wsgiref==0.1.2
.bashrc contains:

export WORKON_HOME=$HOME/Envs
export PROJECT_HOME=$HOME/Projects

~/Envs
~/Projects

Inside hellodjango_venv:

Django==1.5.2
dj-database-url==0.2.2
dj-static==0.0.5
django-toolbelt==0.0.1
gunicorn==18.0
psycopg2==2.5.1
static==0.4

From Terminal:

mac-pol:~ oubiga$ mkdir -p Projects/hellodjango_rep/hellodjango
mac-pol:~ oubiga$ cd Projects/hellodjango_rep/hellodjango
mac-pol:hellodjango oubiga$ mkvirtualenv hellodjango_venv
New python executable in hellodjango_venv/bin/python2.7
Also creating executable in hellodjango_venv/bin/python
Installing Setuptools..................................

...

..................................................done.
(hellodjango_venv)mac-pol:hellodjango oubiga$ pip install django-toolbelt

...

Successfully installed django-toolbelt django psycopg2 gunicorn dj-database-url dj-static static
Cleaning up...
(hellodjango_venv)mac-pol:hellodjango oubiga$ django-admin.py startproject hellodjango .
(hellodjango_venv)mac-pol:hellodjango oubiga$ touch Procfile && open Procfile

Edit Procfile:

web: gunicorn hellodjango.wsgi

Start the process:

(hellodjango_venv)mac-pol:hellodjango oubiga$ foreman start
01:30:37 web.1  | started with pid 638
01:30:37 web.1  | 2013-09-04 01:30:37 [638] [INFO] Starting gunicorn 18.0
01:30:37 web.1  | 2013-09-04 01:30:37 [638] [INFO] Listening at: http://0.0.0.0:5000 (638)
01:30:37 web.1  | 2013-09-04 01:30:37 [638] [INFO] Using worker: sync
01:30:37 web.1  | 2013-09-04 01:30:37 [641] [INFO] Booting worker with pid: 641
CTRL+C

So far, so good.

At the moment my Projects tree is:

~/Projects/    
    hellodjango_rep/
        hellodjango/ cwd
            manage.py
            Procfile
            hellodjango/
                __init__.py
                settings/
                urls.py
                wsgi.py

and my Virtualenvs tree is:

~/Envs/
    get_env_details
    initialize
    postactivate
    postdeactivate
    postmkproject
    postmkvirtualenv
    postrmproject
    postrmvirtualenv
    preactivate
    predeactivate
    premkproject
    premkvirtualenv
    prermproject
    prermvirtualenv
    hellodjango_venv/
        bin/
        include/
        lib/

But, do you remember the three-tiered approach? How can I achieve it?

I am quite sure hellodjango_rep/ later on will contain at least: .git, .gitignore and requirements.txt

Research:

On the #django IRC channel, Jacob Kaplan-Moss answered:

...the Procfile needs to be at the top-level of your git repo...

Neil Middleton, Engineer at Heroku, answered this question "Procfile declares types -> (none) in Heroku" in SO:

...your Procfile needs to be in the root of your git repository that is pushed to Heroku...

From Heroku Dev Center:

...Process types are declared via a file named Procfile placed in the root of your app...

As a First Attempt:

I move Procfile up one folder level.

(hellodjango_venv)mac-pol:hellodjango oubiga$ cd ..

At the moment my Projects tree is:

~/Projects/    
    hellodjango_rep/ cwd
        Procfile
        hellodjango/
            manage.py
            hellodjango/
                __init__.py
                settings/
                urls.py
                wsgi.py

I start the process again:

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ foreman start

I'll get ImportError: No module named hellodjango.wsgi

Debugging:

The ImportError seems to come from /Users/oubiga/Envs/hellodjango_venv/lib/python2.7/site-packages/gunicorn/util.py

def import_app(module):
    parts = module.split(":", 1)
    if len(parts) == 1:
        module, obj = module, "application"
    else:
        module, obj = parts[0], parts[1]

    try:
        __import__(module)  # <-- line 354
    except ImportError:
        if module.endswith(".py") and os.path.exists(module):
            raise ImportError("Failed to find application, did "
                "you mean '%s:%s'?" % (module.rsplit(".", 1)[0], obj))
        else:
            raise

...

No module named hellodjango.wsgi

As a Second Attempt:

Procfile comes back at the previous level. Procfile and manage.py belong together again.

(hellodjango_venv)mac-pol:hellodjango_rep oubiga$ foreman start

I'll get ERROR: Procfile does not exist. It's quite obvious why it happens.

Hypothesis:

From Django documentation:

...manage.py is automatically created in each Django project. manage.py is a thin wrapper around django-admin.py that takes care of two things for you before delegating to django-admin.py:
- It puts your project’s package on sys.path.
- It sets the DJANGO_SETTINGS_MODULE environment variable so that it points to your project’s settings.py file...

I'm not sure if the problem is related to this.

So:

Is it possible to have a Procfile and a manage.py file in different folder level ?
What is wrong with this approach?

Thank you in advance.

Answer 1

Firstly I have to say, that I haven't done so wide research and have used Heroku about 3 times. But:

(Your Seccond Attempt):

Procfile really should be in the top level directory. If you move the Procfile deeper gunicorn can't find it.

(Your First Attempt):

And having your Procfile on the top level directory you should state path to wsgi. Which is not possible yet. So you shoud make __init__.py file in the first "hellodjango" directory, to mark it as "module". Then you shoud change the path in Procfile to: hellodjango.hellodjango.wsgi

Hope this helps.

Answer 2

I ran into this using a Two Scoops layout, and my solution was to keep the standard layout (Procfile in repository_root, manage.py in project_root) and to make Procfile a script that changes directory into project_root before running gunicorn.

e.g., in Procfile:

web: sh -c 'cd ./mysite/ && exec gunicorn mysite.wsgi --log-file -'

You could equally have Procfile call an external script:

web: bash scripts/heroku_run

I prefer this to adding additional module paths as it can mess up imports in settings.py and urls.py, etc.