Confusion regarding Flask & importing modules

Question

I have a flask app that i'm trying to organize and follow the proper flask folder structure and module/importing that's been shown in many tutorials.

I'm at the point where I actually do not know why i'm doing certain things, and that's never a good thing.

My flask app is laid out as follows:

/steam
    run.py
    /steamapp
        __init__.py
        config.py
        tasks.py
        views.py
        /static
           css.css
        /templates
           template.html

run.py

from steamapp import app
app.run()

init.py

from celery import Celery
from config import secrets, constants
from flask.ext.openid import OpenID
from flask import Flask
import praw

def make_celery(app):
    celery = Celery(app.import_name, backend='amqp', broker='amqp://guest@localhost//')
    celery.conf.update(app.config)
    TaskBase = celery.Task
    class ContextTask(TaskBase):
        abstract = True
        def __call__(self, *args, **kwargs):
            with app.app_context():
                return TaskBase.__call__(self, *args, **kwargs)
    celery.Task = ContextTask
    return celery


app = Flask(__name__)
oid = OpenID(app)
app.debug = True
app.secret_key = secrets.APP_SECRET_KEY
celery = make_celery(app)


reddit = praw.Reddit(constants.USERAGENT)
reddit.login(constants.USERNAME, secrets.PASSWORD)

rs = praw.Reddit(constants.USERAGENT)
rs.set_oauth_app_info(secrets.CLIENT_ID, secrets.CLIENT_SECRET, constants.REDIRECT_URL)


from steamapp import views
from steamapp import tasks

views.py

from steamapp import app, oid, celery
from tasks import get_auth_url, check_reddit_oauth, request_steam_api
from flask import Flask, session, redirect, request, render_template, url_for
from config import secrets, constants
from flask.ext.openid import OpenID
import os
import praw
import time
import json
import string
import random
import requests


def id_generator(size=6, chars=string.ascii_uppercase + string.digits):
    return ''.join(random.choice(chars) for _ in range(size))


def check_session_id(session):
    if "id" in session:
        return True
    else:
        return render_template("error.html", error=["?", "No session ID found."])


@app.route("/register")
def register():
    if "id" not in session:
        session["id"] = id_generator()
    authorize_url = get_auth_url(session)
    return redirect(authorize_url)


@app.route("/redirect/")
@oid.loginhandler
def redirect_to_steam_oauth():
   ...
   ...
   ...
   etc

tasks.py

from steamapp import celery, rs
from config import secrets, constants
import requests

@celery.task
def get_auth_url(session):
    return rs.get_authorize_url(session["id"], scope="identity")

When I run run.py I get the following traceback:

Traceback (most recent call last):
  File "/home/andy/Desktop/Python Projects/Finished/steam/run.py", line 1, in <module>
    from steamapp import app
  File "/home/andy/Desktop/Python Projects/Finished/steam/steamapp/__init__.py", line 25, in <module>
    from steamapp import views
  File "/home/andy/Desktop/Python Projects/Finished/steam/steamapp/views.py", line 20, in <module>
    @app.route("/register")
NameError: name 'app' is not defined

I am having trouble understanding what is going on as a whole - I need an explanation for why i'm doing the things i'm doing:

1. What is reason for seperating out app.run() into a seperate python file that imports app from steamapp? Why is the app.run() not in views.py?

2. What is init.py actually meant to do? Why are the views.py and tasks.py imported into it?

3. Why is it giving the traceback of "app" not being defined when it's being imported from __init__.py?

Thank you to anyone who could provide some clarification, as I have lost the sense of what is going on.

---

EDIT:

I've updated the code in the OP with changes i've made after reading @dreamriver's response.

I'm still a bit confused - why am I importing flask in views.py when it is already imported in init.py, where views is imported and presumably ran from?

Also, how does a celery worker tie into this? Setting tasks.py as a Celery worker returns the following traceback:

 celery -A tasks worker --loglevel=info
Traceback (most recent call last):
  File "/usr/local/bin/celery", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 30, in main
    main()
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 769, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 305, in execute_from_commandline
    argv = self.setup_app_from_commandline(argv)
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 465, in setup_app_from_commandline
    self.app = self.find_app(app)
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 485, in find_app
    return find_app(app, symbol_by_name=self.symbol_by_name)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/utils.py", line 229, in find_app
    sym = symbol_by_name(app, imp=imp)
  File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 488, in symbol_by_name
    return symbol_by_name(name, imp=imp)
  File "/usr/local/lib/python2.7/dist-packages/kombu/utils/__init__.py", line 92, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/home/andy/Desktop/Python Projects/Finished/steam/steamapp/tasks.py", line 1, in <module>
    from steamapp import celery, rs
ImportError: No module named steamapp

Answer 1

You have to explicitly import app into your views file. The error says that app is not defined in views.py, that's because you need to do:

from steamapp import app

at the top of your views file.

The __init__.py turns your directory into a python package. At interpreter startup all the directories that are on $PYTHONPATH with __init__.py are added to sys.path which makes them importable. __init__.py is also executed when the module it defines is imported. This makes it extremely useful for e.g. exporting the main apis of your code. If you looked at the source code for flask you would see that all of the code that you do

from flask import Flask, request etc

is actually defined in smaller chunks of functionality in separate files and the interesting pieces are then exposed in __init__.py.

As said here Python doesn't want modules in packages to be the startup file. Python packaging is a bit of a mess. This stackoverflow answer helped me understand some of these questions, namely that relative imports break completely since they are calculated with respect __name__ which is set to '__main__' when you execute the file directly but the filename itself when it is used via import.

Aside from a few module level globals like __name__ and __package__ nothing is explicitly imported into the top level namespace. That's why app isn't available to you in your views.py file implicitly.

Does that answer your questions?

EDIT

You need to import views.py into __init__.py because otherwise your views.py file won't get executed and none of your routes and such will be defined. You need to import app into views.py because app isn't in the namespace that your views.py file is able to access. This pattern where two files import each other is called a circular import and can be tricky but it's fine here. You should know that after a module is loaded it gets cached so it doesn't get re-executed when it is imported again.

Your celery problem looks to me that python doesn't see your app on sys.path when celery is started. Showing the output of sys.path at that point would be helpful. My guess is if you add your working directory to $PYTHONPATH the issue will be fixed. When you install things with pip and the like the packages are added to a place where python knows how to find them by default.