Lambdas inside list comprehensions

Question

I wanted to have a list of lambdas that act as sort of a cache to some heavy computation and noticed this:

>>> [j() for j in [lambda:i for i in range(10)]]
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

Although

>>> list([lambda:i for i in range(10)])
[<function <lambda> at 0xb6f9d1ec>, <function <lambda> at 0xb6f9d22c>, <function <lambda> at 0xb6f9d26c>, <function <lambda> at 0xb6f9d2ac>, <function <lambda> at 0xb6f9d2ec>, <function <lambda> at 0xb6f9d32c>, <function <lambda> at 0xb6f9d36c>, <function <lambda> at 0xb6f9d3ac>, <function <lambda> at 0xb6f9d3ec>, <function <lambda> at 0xb6f9d42c>]

Meaning that the lambdas are unique functions but they somehow all share the same index value.

Is this a bug or a feature? How do I avoid this problem? It's not limited to list comprehensions...

>>> funcs = []
... for i in range(10):
...     funcs.append(lambda:i)
... [j() for j in funcs]
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

Answer 1

What you're seeing here is the effect of closures. The lambda is capturing state from the program to be used later. So while each lambda is a unique object, the state isn't necessarily unique.

The actual 'gotchya' here, is that the variable i is captured, not the value that i represents at that point in time. We can illustrate this with a much easier example:

>>> y = 3
>>> f = lambda: y
>>> f()
3
>>> y = 4
>>> f()
4

The lambda holds on to the reference to the variable, and evaluates that variable when you execute the lambda.

To work around this, you can assign to a local variable within the lambda:

>>> f = lambda y=y:y
>>> f()
4
>>> y = 6
>>> f()
4

Finally, in the case of a loop, the loop variable is only 'declared' once. Therefore, any references to the loop variable within the loop will persist past the next iteration. This includes the variable in list comprehensions.

Answer 2

The lambda returns the value of i at the time you call it. Since you call the lambda after the loop has finished running, the value of i will always be 9.

You can create a local i variable in the lambda to hold the value at the time the lambda was defined:

>>> [j() for j in [lambda i=i:i for i in range(10)]]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Another solution is to create a function that returns the lambda:

def create_lambda(i):
    return lambda:i
>>> [j() for j in [create_lambda(i) for i in range(10)]]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

This works because there is a different closure (holding a different value of i) created for each invocation of create_lambda.

Answer 3

The problem is that you're not capturing the value of i on each iteration of the list comprehension, you're capturing the variable each time through.

The problem is that a closure captures variables by reference. In this case you are capturing a variable whose value changes over time (as with all loop variables), so it has a different value when you run it than when you created it.