Menu
List comprehension is an elegant way to define and create lists based on existing lists. List comprehension is generally more compact and faster than normal functions and loops for creating list. However, we should avoid writing very long list comprehensions in one line to ensure that code is user-friendly.
But why set comprehension was called so? "comprehension" is used here to mean "complete inclusion" or "complete description". A set-comprehension is a (usually short) complete description of a set, not an exhaustive (and possibly infinite) enumeration.
List comprehension offers a shorter syntax when you want to create a new list based on the values of an existing list. Example: Based on a list of fruits, you want a new list, containing only the fruits with the letter "a" in the name.
List comprehensions are useful and can help you write elegant code that's easy to read and debug, but they're not the right choice for all circumstances. They might make your code run more slowly or use more memory.
List comprehensions leak the loop control variable in Python 2 but not in Python 3. Here's Guido van Rossum (creator of Python) explaining the history behind this:
We also made another change in Python 3, to improve equivalence between list comprehensions and generator expressions. In Python 2, the list comprehension "leaks" the loop control variable into the surrounding scope:
x = 'before' a = [x for x in 1, 2, 3] print x # this prints '3', not 'before'This was an artifact of the original implementation of list comprehensions; it was one of Python's "dirty little secrets" for years. It started out as an intentional compromise to make list comprehensions blindingly fast, and while it was not a common pitfall for beginners, it definitely stung people occasionally. For generator expressions we could not do this. Generator expressions are implemented using generators, whose execution requires a separate execution frame. Thus, generator expressions (especially if they iterate over a short sequence) were less efficient than list comprehensions.
However, in Python 3, we decided to fix the "dirty little secret" of list comprehensions by using the same implementation strategy as for generator expressions. Thus, in Python 3, the above example (after modification to use print(x) :-) will print 'before', proving that the 'x' in the list comprehension temporarily shadows but does not override the 'x' in the surrounding scope.
Yes, list comprehensions "leak" their variable in Python 2.x, just like for loops.
In retrospect, this was recognized to be a mistake, and it was avoided with generator expressions. EDIT: As Matt B. notes it was also avoided when set and dictionary comprehension syntaxes were backported from Python 3.
List comprehensions' behavior had to be left as it is in Python 2, but it's fully fixed in Python 3.
This means that in all of:
list(x for x in a if x>32)
set(x//4 for x in a if x>32) # just another generator exp.
dict((x, x//16) for x in a if x>32) # yet another generator exp.
{x//4 for x in a if x>32} # 2.7+ syntax
{x: x//16 for x in a if x>32} # 2.7+ syntax
the x is always local to the expression while these:
[x for x in a if x>32]
set([x//4 for x in a if x>32]) # just another list comp.
dict([(x, x//16) for x in a if x>32]) # yet another list comp.
in Python 2.x all leak the x variable to the surrounding scope.
UPDATE for Python 3.8(?): PEP 572 will introduce := assignment operator that deliberately leaks out of comprehensions and generator expressions! It's motivated by essentially 2 use cases: capturing a "witness" from early-terminating functions like any() and all():
if any((comment := line).startswith('#') for line in lines):
print("First comment:", comment)
else:
print("There are no comments")
and updating mutable state:
total = 0
partial_sums = [total := total + v for v in values]
See Appendix B for exact scoping. The variable is assigned in closest surrounding def or lambda, unless that function declares it nonlocal or global.
Yes, assignment occurs there, just like it would in a for loop. No new scope is being created.
This is definitely the expected behavior: on each cycle, the value is bound to the name you specify. For instance,
>>> x=0
>>> a=[1,54,4,2,32,234,5234,]
>>> [x for x in a if x>32]
[54, 234, 5234]
>>> x
5234
Once that's recognized, it seems easy enough to avoid: don't use existing names for the variables within comprehensions.
Interestingly this doesn't affect dictionary or set comprehensions.
>>> [x for x in range(1, 10)]
[1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> x
9
>>> {x for x in range(1, 5)}
set([1, 2, 3, 4])
>>> x
9
>>> {x:x for x in range(1, 100)}
{1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6, 7: 7, 8: 8, 9: 9, 10: 10, 11: 11, 12: 12, 13: 13, 14: 14, 15: 15, 16: 16, 17: 17, 18: 18, 19: 19, 20: 20, 21: 21, 22: 22, 23: 23, 24: 24, 25: 25, 26: 26, 27: 27, 28: 28, 29: 29, 30: 30, 31: 31, 32: 32, 33: 33, 34: 34, 35: 35, 36: 36, 37: 37, 38: 38, 39: 39, 40: 40, 41: 41, 42: 42, 43: 43, 44: 44, 45: 45, 46: 46, 47: 47, 48: 48, 49: 49, 50: 50, 51: 51, 52: 52, 53: 53, 54: 54, 55: 55, 56: 56, 57: 57, 58: 58, 59: 59, 60: 60, 61: 61, 62: 62, 63: 63, 64: 64, 65: 65, 66: 66, 67: 67, 68: 68, 69: 69, 70: 70, 71: 71, 72: 72, 73: 73, 74: 74, 75: 75, 76: 76, 77: 77, 78: 78, 79: 79, 80: 80, 81: 81, 82: 82, 83: 83, 84: 84, 85: 85, 86: 86, 87: 87, 88: 88, 89: 89, 90: 90, 91: 91, 92: 92, 93: 93, 94: 94, 95: 95, 96: 96, 97: 97, 98: 98, 99: 99}
>>> x
9
However it has been fixed in 3 as noted above.
some workaround, for python 2.6, when this behaviour is not desirable
# python
Python 2.6.6 (r266:84292, Aug 9 2016, 06:11:56)
Type "help", "copyright", "credits" or "license" for more information.
>>> x=0
>>> a=list(x for x in xrange(9))
>>> x
0
>>> a=[x for x in xrange(9)]
>>> x
8
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With