Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Nested list comprehension scope

Tags:

The best way to explain my question is with an example:

example.py:

class A(object):     integers = [1, 2, 3]     singles = [i for i in integers]  class B(object):     integers = [1, 2, 3]     pairs = [(i, j) for i in integers for j in integers] 

When I run this under python 2 it works fine, but under python 3 I get a NameError for class B (but not class A):

$ python example.py Traceback (most recent call last):   File "example.py", line 6, in <module>     class B(object):   File "example.py", line 8, in B     pairs = [(i, j) for i in integers for j in integers]   File "example.py", line 8, in <listcomp>     pairs = [(i, j) for i in integers for j in integers] NameError: global name 'integers' is not defined 

Why does only class B raise a NameError and why only under Python 3?

like image 806
sn6uv Avatar asked Nov 22 '13 04:11

sn6uv


People also ask

Can you do nested list comprehension?

As it turns out, you can nest list comprehensions within another list comprehension to further reduce your code and make it easier to read still. As a matter of fact, there's no limit to the number of comprehensions you can nest within each other, which makes it possible to write very complex code in a single line.

Is list comprehension faster than appending?

Even factoring out the time it takes to lookup and load the append function, the list comprehension is still faster because the list is created in C, rather than built up one item at a time in Python.

What is list comprehension and how is it useful?

List comprehension is an elegant way to define and create lists based on existing lists. List comprehension is generally more compact and faster than normal functions and loops for creating list. However, we should avoid writing very long list comprehensions in one line to ensure that code is user-friendly.

What is the advantage of list comprehension in Python?

One main benefit of using a list comprehension in Python is that it's a single tool that you can use in many different situations. In addition to standard list creation, list comprehensions can also be used for mapping and filtering. You don't have to use a different approach for each scenario.


1 Answers

Class scopes are a bit strange in Python 3, but it's for a good reason.

In Python 2, the iteration variables (i and j in your examples) leaked out of list comprehensions and would be included in the outside scope. This is because they were developed early in Python 2's design, and they were based on explicit loops. As an example of how this is unexpected, check the values of B.i and B.j in Python 2 where you didn't get an error!

In Python 3, list comprehensions were changed to prevent this leaking. They are now implemented with a function (which has its own scope) that is called to produce the list value. This makes them work the same as generator expressions, which have always been functions under the covers.

A consequence of this is that in a class, a list comprehension usually can't see any class variables. This is parallel to a method not being able to see class variables directly (only though self or the explicit class name). For example, calling the method in the class below will give the same NameError exception you are seeing in your list comprehension:

class Foo:     classvar = "bar"     def blah(self):         print(classvar) # raises "NameError: global name 'classvar' is not defined" 

There is an exception, however: The sequence being iterated over by the first for clause of a list comprehension is evaluated outside of the inner function. This is why your A class works in Python 3. It does this so that generators can catch non-iterable objects immediately (rather than only when next is called on them and their code runs).

But it doesn't work for the inner for clause in the two-level comprehension in class B.

You can see the difference if you disassemble some functions that create list comprehensions using the dis module:

def f(lst):     return [i for i in lst]  def g(lst):     return [(i, j) for i in lst for j in lst] 

Here's the disassembly of f:

>>> dis.dis(f)   2           0 LOAD_CONST               1 (<code object <listcomp> at 0x0000000003CCA1E0, file "<pyshell#374>", line 2>)                3 LOAD_CONST               2 ('f.<locals>.<listcomp>')                6 MAKE_FUNCTION            0                9 LOAD_FAST                0 (lst)               12 GET_ITER                           13 CALL_FUNCTION            1 (1 positional, 0 keyword pair)               16 RETURN_VALUE        

The first three lines show f loading up a precompiled code block and creating a function out of it (it names it f.<locals>.<listcomp>). This is the function used to make the list.

The next two lines show the lst variable being loaded and an iterator being made from it. This is happening within f's scope, not the inner function's. Then the <listcomp> function is called with that iterator as its argument.

This is comparable to class A. It gets the iterator from the class variable integers, just like you can use other kinds of references to previous class members in the definition of a new member.

Now, compare the disassembly of g, which makes pairs by iterating over the same list twice:

>>> dis.dis(g)   2           0 LOAD_CLOSURE             0 (lst)                3 BUILD_TUPLE              1                6 LOAD_CONST               1 (<code object <listcomp> at 0x0000000003CCA810, file "<pyshell#377>", line 2>)                9 LOAD_CONST               2 ('g.<locals>.<listcomp>')               12 MAKE_CLOSURE             0               15 LOAD_DEREF               0 (lst)               18 GET_ITER                           19 CALL_FUNCTION            1 (1 positional, 0 keyword pair)               22 RETURN_VALUE          

This time, it builds a closure with the code object, rather than a basic function. A closure is a function with some "free" variables that refer to things in the enclosing scope. For the <listcomp> function in g, this works just fine, since its scope is a normal one. However, when you try to use the same sort of comprehension in class B, the closure fails, since classes don't let functions they contain see into their scopes in that way (as demonstrated with the Foo class above).

It's worth noting that not only inner sequence values cause this issue. As in the previous question linked to by BrenBarn in a comment, you'll have the same issue if a class variable is referred to elsewhere in the list comprehension:

class C:     num = 5     products = [i * num for i in range(10)] # raises a NameError about num 

You don't, however, get an error from multi-level list comprehensions where the inner for (or if) clauses only refer to the results of the preceding loops. This is because those values aren't part of a closure, just local variables inside the <listcomp> function's scope.

class D:     nested = [[1, 2, 3], [4, 5, 6]]     flattened = [item for inner in nested for item in inner] # works! 

Like I said, class scopes are a bit strange.

like image 67
Blckknght Avatar answered Sep 22 '22 19:09

Blckknght