Why do list comprehensions write to the loop variable, but generators don't? [duplicate]

Question

If I do something with list comprehensions, it writes to a local variable:

i = 0 test = any([i == 2 for i in xrange(10)]) print i 

This prints "9". However, if I use a generator, it doesn't write to a local variable:

i = 0 test = any(i == 2 for i in xrange(10)) print i 

This prints "0".

Is there any good reason for this difference? Is this a design decision, or just a random byproduct of the way that generators and list comprehensions are implemented? Personally, it would seem better to me if list comprehensions didn't write to local variables.

Answer 1

As PEP 289 (Generator Expressions) explains:

The loop variable (if it is a simple variable or a tuple of simple variables) is not exposed to the surrounding function. This facilitates the implementation and makes typical use cases more reliable.

It appears to have been done for implementation reasons.

Personally, it would seem better to me if list comprehensions didn't write to local variables.

PEP 289 clarifies this as well:

List comprehensions also "leak" their loop variable into the surrounding scope. This will also change in Python 3.0, so that the semantic definition of a list comprehension in Python 3.0 will be equivalent to list().

In other words, the behaviour you describe indeed differs in Python 2 but it has been fixed in Python 3.

Answer 2

Python’s creator, Guido van Rossum, mentions this when he wrote about generator expressions that were uniformly built into Python 3: (emphasis mine)

We also made another change in Python 3, to improve equivalence between list comprehensions and generator expressions. In Python 2, the list comprehension "leaks" the loop control variable into the surrounding scope:

x = 'before' a = [x for x in 1, 2, 3] print x # this prints '3', not 'before' 

This was an artifact of the original implementation of list comprehensions; it was one of Python's "dirty little secrets" for years. It started out as an intentional compromise to make list comprehensions blindingly fast, and while it was not a common pitfall for beginners, it definitely stung people occasionally. For generator expressions we could not do this. Generator expressions are implemented using generators, whose execution requires a separate execution frame. Thus, generator expressions (especially if they iterate over a short sequence) were less efficient than list comprehensions.

However, in Python 3, we decided to fix the "dirty little secret" of list comprehensions by using the same implementation strategy as for generator expressions. Thus, in Python 3, the above example (after modification to use print(x) :-) will print 'before', proving that the 'x' in the list comprehension temporarily shadows but does not override the 'x' in the surrounding scope.

So in Python 3 you won’t see this happen anymore.

Interestingly, dict comprehensions in Python 2 don’t do this either; this is mostly because dict comprehensions were backported from Python 3 and as such already had that fix in them.

There are some other questions that cover this topic too, but I’m sure you have already seen those when you searched for the topic, right? ;)

• Python list comprehension rebind names even after scope of comprehension. Is this right?
• Why the list comprehension variable is accessible after the operation is done?