Captured variables in "eval" in Python

Question

I'm having trouble understanding the semantics of "eval()" and "exec" in Python. (All code in this question behaves the same way in Python 2.7.8 and Python 3.4.2). The documentation for "eval" says:

If both [locals and globals] are omitted, the expression is executed in the environment where eval() is called.

There is similar language for "exec". I clearly don't understand this sentence because I would expect the four functions defined by the following program to do the same thing.

def h(x):
    ls = locals()
    exec('def i(y): return (w, x, y)', globals(), ls)
    i = ls['i']
    def       j(y): return (w, x, y)
    k = eval('lambda y: (w, x, y)')
    l =       lambda y: (w, x, y)
    return i, j, k, l

w = 1

i, j, k, l = h(2)

They do not.

>>> i(3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in i
NameError: name 'x' is not defined
>>> j(3)
(1, 2, 3)
>>> k(3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <lambda>
NameError: name 'x' is not defined
>>> l(3)
(1, 2, 3)

Disassembling the code reveals why: "x" is treated as a global variable by "eval" and "exec".

from dis import dis
print("This is `i`:")
dis(i)
print("This is `j`:")
dis(j)
print("This is `k`:")
dis(k)
print("This is `l`:")
dis(l)
print("For reference, this is `h`:")
dis(h)

Output:

This is `i`:
  1           0 LOAD_GLOBAL              0 (w)
              3 LOAD_GLOBAL              1 (x)
              6 LOAD_FAST                0 (y)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE        
This is `j`:
 25           0 LOAD_GLOBAL              0 (w)
              3 LOAD_DEREF               0 (x)
              6 LOAD_FAST                0 (y)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE        
This is `k`:
  1           0 LOAD_GLOBAL              0 (w)
              3 LOAD_GLOBAL              1 (x)
              6 LOAD_FAST                0 (y)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE        
This is `l`:
 27           0 LOAD_GLOBAL              0 (w)
              3 LOAD_DEREF               0 (x)
              6 LOAD_FAST                0 (y)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE        
For reference, this is `h`:
 22           0 LOAD_NAME                0 (locals)
              3 CALL_FUNCTION            0
              6 STORE_FAST               1 (ls)

 23           9 LOAD_CONST               1 ('def i(y): return (w, x, y)')
             12 LOAD_NAME                1 (globals)
             15 CALL_FUNCTION            0
             18 LOAD_FAST                1 (ls)
             21 EXEC_STMT           

 24          22 LOAD_FAST                1 (ls)
             25 LOAD_CONST               2 ('i')
             28 BINARY_SUBSCR       
             29 STORE_FAST               2 (i)

 25          32 LOAD_CLOSURE             0 (x)
             35 BUILD_TUPLE              1
             38 LOAD_CONST               3 (<code object j at 0x7ffc3843c030, file "test.py", line 25>)
             41 MAKE_CLOSURE             0
             44 STORE_FAST               3 (j)

 26          47 LOAD_NAME                2 (eval)
             50 LOAD_CONST               4 ('lambda y: (w, x, y)')
             53 CALL_FUNCTION            1
             56 STORE_FAST               4 (k)

 27          59 LOAD_CLOSURE             0 (x)
             62 BUILD_TUPLE              1
             65 LOAD_CONST               5 (<code object <lambda> at 0x7ffc3843c3b0, file "test.py", line 27>)
             68 MAKE_CLOSURE             0
             71 STORE_FAST               5 (l)

 28          74 LOAD_FAST                2 (i)
             77 LOAD_FAST                3 (j)
             80 LOAD_FAST                4 (k)
             83 LOAD_FAST                5 (l)
             86 BUILD_TUPLE              4
             89 RETURN_VALUE

The question

"j" and "l" above have the behaviour I want. How can I get this behaviour using "eval" or "exec"?

Failure 1

Using a class instead of a function as the outer wrapper does change the semantics, but in the opposite to the desired way. It makes "x" into a global.

class H:
    x = 2
    f = staticmethod(eval('lambda y: (w, x, y)'))

H.dis(H.f)

w = 1
H.f(3)

Output:

  1           0 LOAD_GLOBAL              0 (w)
              3 LOAD_GLOBAL              1 (x)
              6 LOAD_FAST                0 (y)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE        
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <lambda>
NameError: global name 'x' is not defined

Wrapping in "classmethod" or leaving it as an unbound instance method just makes things worse.

Failure 2

Substituting "x" using string interpolation works for integers:

def h(x):
    return eval('lambda y: (w, %r, y)' % x)

k = h(2)

dis(k)

w = 1
k(3)

Output:

  1           0 LOAD_GLOBAL              0 (w)
              3 LOAD_CONST               1 (2)
              6 LOAD_FAST                0 (y)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE        
(1, 2, 3)

However, I don't want to assume that "x" can be losslessly converted to a string and back. The attempt is broken in the following examples:

k = h(lambda: "something")

k = h(open('some_file', 'w'))

cell = ["Wrong value"]
k = h(cell)
cell[0] = "Right value"
k(3)

Failure 3

Since Python is looking for a global variable, one obvious attempt is to pass "x" as a global variable:

def h(x):
    my_globals = {'w': w, 'x': x}
    return eval('lambda y: (w, x, y)', my_globals)

k = h(2)

dis(k)

w = 1
k(3)

Output:

  1           0 LOAD_GLOBAL              0 (w)
              3 LOAD_GLOBAL              1 (x)
              6 LOAD_FAST                0 (y)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE        
(1, 2, 3)

This attempt is broken because it reads the value of "w" too early:

w = "Wrong value"
k = h(2)
w = "Right value"
k(3)

Success 1

I did eventually find an approach that works, but I really don't like it:

def h(x):
    return eval('lambda x: lambda y: (w, x, y)')(x) 

k = h(2)

dis(k)

w = 1
k(3)

Output:

  1           0 LOAD_GLOBAL              0 (w)
              3 LOAD_DEREF               0 (x)
              6 LOAD_FAST                0 (y)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE
(1, 2, 3)

In particular, this is going to become painful if I do not know the full list of local variables captured by the string I'm passing to "eval".

Can you do better?

Update 2014-12-25

Failure 4

Looking for more ways of creating the local variable "x", I tried this:

def h(x):
    ls = locals()
    exec('x = x\ndef i(y): return (w, x, y)', globals(), ls)
    exec('_ = x\ndef j(y): return (w, x, y)', globals(), ls)
    return ls['i'], ls['j'], ls['_'], ls['x']

i, j, check1, check2 = h(2)

assert check1 == 2
assert check2 == 2

w = 1

print("This is `i`:")
dis(i)
print("This is `j`:")
dis(j)

print("i(3) = %r" % (i(3),))
print("j(3) = %r" % (j(3),))

The extra assignment to "x" has no effect. The assertions verify that "x" is in the dictionary of locals, but it is not captured by the lambdas. Here is the output:

This is `i`:
  2           0 LOAD_GLOBAL              0 (w)
              3 LOAD_GLOBAL              1 (x)
              6 LOAD_FAST                0 (y)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE
This is `j`:
  2           0 LOAD_GLOBAL              0 (w)
              3 LOAD_GLOBAL              1 (x)
              6 LOAD_FAST                0 (y)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE

The calls to "i" and "j" both crash, complaining that there is no global variable "x".

Success 2

[Edit 2014-12-29: This succeeds only on Python 3.]

Another way of creating a local variable is like this:

def h(x):
    i = eval('[lambda y: (w, x, y) for x in [x]][0]')
    j = eval('[lambda y: (w, x, y) for _ in [x]][0]')
    return i, j

i, j = h(2)

w = 1

print("This is `i`:")
dis(i)
print("This is `j`:")
dis(j)

print("i(3) = %r" % (i(3),))
print("j(3) = %r" % (j(3),))

Strangely, in this case the extra assignment to "x" does have an effect. This does work, i.e. "i" is different from "j". Here is the output:

This is `i`:
  1           0 LOAD_GLOBAL              0 (w)
              3 LOAD_DEREF               0 (x)
              6 LOAD_FAST                0 (y)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE
This is `j`:
  1           0 LOAD_GLOBAL              0 (w)
              3 LOAD_GLOBAL              1 (x)
              6 LOAD_FAST                0 (y)
              9 BUILD_TUPLE              3
             12 RETURN_VALUE
i(3) = (1, 2, 3)

The call to "j" crashes, complaining that there is no global "x", but "i" works as desired and has the correct bytecode.

Why does this work, while "Failure 4" above does not? What is the rule that determines whether the local "x" can be captured? And what is the history of this design? (It seems absurd!)

Answer 1

I think you want your created functions to inherit the local environment of the function that creates them, but also the real global environment (of the function that creates them). That's why you don't like them referring to x as a global, right?

The following creates a "wrapper" function around the desired function, all within the same exec string. The values of the creating function's locals are passed in when you call or re-call the wrapper, creating a new wrapped closure.

The code is sensitive to new variables being created in the locals context. It goes to some trouble to make sure that the function and wrapper names are both known and have values there.

def wrap_f(code, gs, ls, wrapper_name, function_name):
    ls[function_name] = ls[wrapper_name] = None
    arglist = ",".join(ls.keys())
    wrapcode = """
def {wrapper_name}({arglist}):
{code}
    return {function_name}
    """.format(wrapper_name=wrapper_name, arglist=arglist, 
               code=code, function_name=function_name)
    exec(wrapcode, gs, ls)
    wrapper = ls[wrapper_name]
    return wrapper, wrapper(**ls)

So, to answer the original question, this code...

def h(x):
    mcode = "    def m(y): return (w, x, y)"  # must be indented 4 spaces.
    mwrap, m = wrap_f(mcode, globals(), locals(), "mwrap", "m")
    return m

w = 1
m = h(2)
print m(3)

...produces this output:

(1, 2, 3)

And this example shows what to do when the locals in the creator function change:

def foo(x):
    barleycode = """
    def barley(y):
        print "barley's x =", x
        print "barley's y =", y
    """
    barleywrap, barley = wrap_f(barleycode, globals(), locals(), 
                               "barleywrap", "barley")
    barley("this string")
    print

    x = "modified x"
    barley = barleywrap(**locals())
    barley("new arg")
    print

    x = "re-modified x"
    barley("doesn't see the re-mod.")

x = "the global x"

foo("outer arg")

This produces the output:

barley's x = outer arg
barley's y = this string

barley's x = modified x
barley's y = new arg

barley's x = modified x
barley's y = doesn't see the re-mod.