list comprehension in exec with empty locals: NameError

Question

Consider the following snippet:

def bar():
    return 1
print([bar() for _ in range(5)])

It gives an expected output [1, 1, 1, 1, 1].

However, if I try to exec the same snippet in empty environment (locals and globals both are set to {}), it gives NameError:

if 'bar' in globals() or 'bar' in locals():
    del bar
# make sure we reset settings

exec("""
def bar():
    return 1
print([bar() for _ in range(5)])
""", {}, {})

NameError: name 'bar' is not defined

If I invoke exec like exec(…, {}) or exec(…), it is executed as expected.

Why?

EDIT:

Consider also the following snippet:

def foo():
    def bar():
        return 1
    print('bar' in globals()) # False
    print('bar' in locals()) # True
    print(['bar' in locals() for _ in [1]]) # [False]
    print([bar() for _ in [1, 2]]) # [1, 1]

Just like in my first exec, we don't have bar in locals inside list comprehension. However, if we try to invoke it, it works!

Answer 1

The solution to your problem lies here:

In all cases, if the optional parts are omitted, the code is executed in the current scope. If only globals is provided, it must be a dictionary, which will be used for both the global and the local variables. If globals and locals are given, they are used for the global and local variables, respectively. If provided, locals can be any mapping object. Remember that at module level, globals and locals are the same dictionary. If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.

https://docs.python.org/3/library/functions.html#exec

Basically, your problem is that bar is defined in the scope of locals and only in locals. Therefore, this exec() statement works:

exec("""
def bar():
    return 1
print(bar())
""", {}, {})

The list comprehension however creates a new local scope, one in which bar is not defined and can therefore not be looked up.

This behaviour can be illustrated with:

exec("""
def bar():
    return 1
print(bar())
print(locals())
print([locals() for _ in range(1)])
""", {}, {})

which returns

1
{'bar': <function bar at 0x108efde18>}
[{'_': 0, '.0': <range_iterator object at 0x108fa8780>}]

EDIT

In your original example, the definition of bar is found in the (module level) global scope. This corresponds to

Remember that at module level, globals and locals are the same dictionary.

In the exec example, you introduce an artificial split in scopes between globals and locals by passing two different dictionaries. If you passed the same one or only the globals one (which would in turn mean that this one will be used for both globals and locals) , your example would also work.

As for the example introduced in the edit, this boils down to the scoping rules in python. For a detailed explanation, please read: https://docs.python.org/3/tutorial/classes.html#python-scopes-and-namespaces

In short, while bar is not in the local scope of the list comprehension and neither in the global scope, it is in the scope of foo. And given Python scoping rules, if a variable is not found in the local scope, it will be searched for in the enclosing scopes until the global scope is reached. In your example, foo's scope sits between the local scope and the global scope, so bar will be found before reaching the end of the search.

This is however still different to the exec example, where the locals scope you pass in is not enclosing the scope of the list comprehension, but completely divided from it.

Another great explanation of scoping rules including illustrations can be found here: http://sebastianraschka.com/Articles/2014_python_scope_and_namespaces.html

Answer 2

As Hendrik Makait found out, the exec documentation says that

If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.

You can get the same behaviour by embedding the code into a class definition:

class Foo:
    def bar():
        return 1
    print([bar() for _ in range(5)])

Run it in Python 3 and you will get

Traceback (most recent call last):
  File "foo.py", line 9, in <module>
    class Foo:
  File "foo.py", line 15, in Foo
    print({bar() for _ in range(5)})
  File "foo.py", line 15, in <setcomp>
    print({bar() for _ in range(5)})
NameError: global name 'bar' is not defined

The reason for the error is as Hendrik said that a new implicit local scope is created for list comprehensions. However Python only ever looks names up in 2 scopes: global or local. Since neither the global nor the new local scope contains the name bar, you get the NameError.

The code works in Python 2, because list comprehensions have a bug in Python 2 in that they do not create a new scope, and thus they leak variables into their current local scope:

class Foo:
    [1 for a in range(5)]
    print(locals()['a'])

Run it in Python 2 and the output is 4. The variable a is now within the locals in the class body, and retains the value from the last iteration. In Python 3 you will get a KeyError.

You can get the same error in Python 2 too though, if you use a generator expression, or a dictionary/set comprehension:

class Foo:
    def bar():
        return 1
    print({bar() for _ in range(5)})

The error can be produced also by just using simply

class Foo: 
    bar = 42
    class Bar:
        print(bar)

This is unlike

def foo():
    bar = 42
    def baz():
        print(bar)
    baz()

because upon execution of foo, Python makes baz into a closure, which will access the bar variable via a special bytecode instruction.