Why doesn't exec work in a function with a subfunction?

Question

Correct. You can't use exec in a function that has a subfunction, unless you specify a context. From the docs:

If exec is used in a function and the function contains a nested block with free variables, the compiler will raise a SyntaxError unless the exec explicitly specifies the local namespace for the exec. (In other words, "exec obj" would be illegal, but "exec obj in ns" would be legal.)

There is good reason for this which I would probably understand if it wasn't Sunday night. Now, next question: Why are you using exec? It's very rarely needed. You say you have a good reason. I'm feeling sceptical about that. ;) If you have a good reason I'll tell you the workaround. :-P

Oh well, here it is anyway:

def test2():
    """Test with a subfunction."""
    exec 'print "hi from test2"' in globals(), locals()
    def subfunction():
        return True

Although in Python it looks kind of like the local variables are stored in a dictionary locals(), they usually aren't. Instead they are mostly stored on the stack and accessed by index. This makes local variable lookup faster than if it had to do a dictionary lookup every time. If you use the locals() function then what you get is a fresh dictionary created from all the local variables, and that's why assigning to locals() doesn't generally work.

There are a couple of exceptions to this scenario:

When you use an unqualified exec inside a function Python turns off the optimisation and uses a real dictionary for the local variables. That means you can create or update variables from inside the exec, but it also means all local variable access in that function will run more slowly.

The other exception is that when you nest functions the inner function can access local variables in the outer function scope. When it does this the variable is stored in a 'cell' object instead of being stored on the stack. The extra level of indirection makes all use of scoped variables slower whether you access them from the inner or outer function.

The catch that you've encountered is that these two exceptions to how local variables are normally stored are incompatible. You cannot have a variable stored in a dictionary and accessed through a cell reference at the same time. Python 2.x fixes this by disallowing the exec, even in cases like this where you aren't trying to use any scoped variables.

This is a rather interesting case:

>>> def func():
...     exec('print "hi from func"')
...     def subfunction():
...         return True
... 
  File "<stdin>", line 2
SyntaxError: unqualified exec is not allowed in function 'func' because 
it contains a nested function with free variables

The reason why this doesn't work indeed is that subfunction contains a free variable, and since in Python 2, exec could theoretically modify the locals in the containing scope, it would be impossible to decide if the variable should be bound from the global or the parent function scope. One of the verses in the Zen of Python is "In the face of ambiguity, refuse the temptation to guess." and this is what Python 2 does.

Now the question is: what is this free (unbound) variable? Well, it is True!

Indeed it is reproduceable with None:

>>> def func():
...     exec('print "hi from func"')
...     def subfunction():
...         return None
... 
  File "<stdin>", line 2
SyntaxError: unqualified exec is not allowed in function 'test2' because it contains a nested
function with free variables

Even though None cannot be assigned to, and it is considered as a constant in the bytecode, the buggy parser thinks it is an unbound variable here.

But if you replace it with 1 and it works without problems:

>>> def test2():
...     exec('print "hi from func"')
...     def subfunction():
...         return 1
... 
>>>

To avoid this error, specify explicitly the globals and possibly locals that are to be used by exec, say:

>>> def test2():
...     exec 'print "hi from test2"' in {}
...     def subfunction():
...         return None
...
>>>

---

In Python 3, exec is just a simple function and isn't handled specially by the parser or the bytecode compiler. In Python 3 exec cannot rebind function-local names, and thus this SyntaxError and ambiguity doesn't exist.

---

One peculiar case in Python 2 vs 3 compatibility is that while Python 2.7 documentation states that

The form exec(expr, globals) is equivalent to exec expr in globals, while the form exec(expr, globals, locals) is equivalent to exec expr in globals, locals. The tuple form of exec provides compatibility with Python 3, where exec is a function rather than a statement.

The tuple form has not always been 100 % compatible, as there was a bug in handling of exec in functions with nested functions (issue 21591); up to Python 2.7.8 the following code might have thrown an exception:

def func():
    exec('print "hi from test2"', {})
    def subfunction():
        return None

This was fixed in Python 2.7.9 and it no longer throws.

That works well in Python 3.1.3, after modifying the print statement to use print function.

In Python 2.6, it produces SyntaxError: unqualified exec is not allowed in function 'test2' it contains a nested function with free variables, I don't think it's a bug.

dict and list comprehensions may also be considered subfunctions on Python 2.7.5

For example, this fails on Python 2.7.5, but works on Python 2.7.12:

def func():
    exec('print("a")')
    (str(e) for e in range(10))

with:

  File "./a.py", line 4
    exec('print("a")')
SyntaxError: unqualified exec is not allowed in function 'func' it contains a nested function with free variables

Likely it got internally compiled to a function in the bytecode.

TODO find the fixing commit. It was beyond my git log --grep foo.

Analogous for dict comprehensions:

def func():
    exec('print("a")', {e:str(e) for e in range(10)})

which is specially bad since it is a common parameter for the global argument.

Also raised at: https://github.com/sphinx-doc/sphinx/issues/5417#issuecomment-421731085