python can't access nonlocal variable before local variable is defined with same name [duplicate]

Question

I've used decorators before and so I was surprised to find a bug in my code:

def make_handler(name, panels):
    def get(self):
        admin = True
        keys = [ndb.Key('Panel', panel) for panel in panels]
        panels = zip(ndb.get_multi(keys), panels)
        panels = [(panel.panel_html if panel else get_default_content(panel_id), panel_id) for panel, panel_id in panels]
        templates = {'panels': panels, 'admin': admin}
        self.render_template('panel_page.html', **templates)
    return type(name, (BaseHandler,), {'get': get})

The resulting error is:

Traceback (most recent call last):
  File "C:\Program Files\Google\google_appengine\lib\webapp2\webapp2.py", line 1536, in __call__
    rv = self.handle_exception(request, response, e)
  File "C:\Program Files\Google\google_appengine\lib\webapp2\webapp2.py", line 1530, in __call__
    rv = self.router.dispatch(request, response)
  File "C:\Program Files\Google\google_appengine\lib\webapp2\webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "C:\Program Files\Google\google_appengine\lib\webapp2\webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "C:\Program Files\Google\google_appengine\lib\webapp2\webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "C:\Program Files\Google\google_appengine\lib\webapp2\webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "C:\Users\Robert\PycharmProjects\balmoral_doctors\main.py", line 35, in get
    keys = [ndb.Key('Panel', panel) for panel in panels]
UnboundLocalError: local variable 'panels' referenced before assignment

My fix is to change panel to panel2 beyond the first usage:

def make_handler(name, panels):
    def get(self):
        admin = True
        keys = [ndb.Key('Panel', panel) for panel in panels]
        panels2 = zip(ndb.get_multi(keys), panels)
        panels2 = [(panel.panel_html if panel else get_default_content(panel_id), panel_id) for panel, panel_id in panels2]
        templates = {'panels': panels2, 'admin': admin}
        self.render_template('panel_page.html', **templates)
    return type(name, (BaseHandler,), {'get': get})

This is my understanding: panels = zip(...) means that panels is a local variable, so the function doesn't look in the outer scope for panels.

This is done before the get() function is run, and not midway through?

I thought it would firstly grab panels from the outer function, and then when panels gets defined in the inner function after that, it would from then on use the new local panels variable.

Am I on the right track?

Answer 1

Yes.

Python's scoping rules indicate that a function defines a new scope level, and a name is bound to a value in only one scope level in a scope level—it is statically scoped (i.e. all scoping is determined at compilation time). As you understood, you're trying to violate that by reading from a non-local declaration and writing to a local variable. As you observe, the interpreter objects violently to this by raising an UnboundLocalError: it has understood that panels is a local variable (because it can't be that and non-local at the same time), but you haven't assigned (bound) a value to the name, and so it fails.

More technical details

The decision was made in Python to keep track of where variables are at compilation time in the bytecode (to be specific for this case, it's in a tuple get.__code__.co_varnames for local variables), meaning that a variable can be only used in a single scope level in a certain scope. In Python 2.x, it is not possible to modify a non-local variable; you have either read-only access to a global or non-local variable, or read-write access to a global variable by using the global statement, or read-write access to a local variable (the default). That's just the way it's been designed (probably for performance and purity). In Python 3, the nonlocal statement has been introduced with a similar effect to global, but for an intermediate scope.

Binding the modified variable to a different name is the correct solution in this case.

Answer 2

You are more-or-less correct, and you found the correct resolution. Your problem is equivalent to this:

bars = range(10)

def foo():
    thing = [x for x in bars]
    bars = 'hello'

foo()
# UnboundLocalError: local variable 'bars' referenced before assignment

At function definition time, it is determined that bars is local scope. And then at function run time, you get the problem that bars hasn't been assigned.

Answer 3

Many people don't realise this, but Python is actually statically scoped. When Python sees a read from a bare name (i.e. not an attribute of some object), it can determine precisely where that name read will go to purely from compile-time analysis.

If a name is ever assigned to in a function, then that name is a local variable in that function¹, and it's scope extends over the entirety of the function's body, even over the lines before the assignment.

If the name is not assigned to in a function, then the name is a non-local variable. Python can check the static scope of any surrounding def blocks to see if it's a local variable in any of those. If not, then that name must be a reference to a module global or a built-in (the choice between global and built-in is resolved dynamically, so you can shadow a built-in with a global that is declared dynamically).

I believe this is done mainly for efficiently. It means that the set of local variables of a function can be known when the bytecode for the function is compiled, and local variable access can be converted into simple index operations. Otherwise Python would have to perform dictionary lookups to access local variables, which would be slower.

So because your get function contains a line of the form panels = ..., then panels is a local variable in the entirety of the body of get. The assignment to keys is looping over the local variable panels before it has been assigned.

---

¹ Unless that name is declared global or nonlocal, but that's still statically known.