Unexpected list comprehension behaviour in Python

Question

I believe I'm getting bitten by some combination of nested scoping rules and list comprehensions. Jeremy Hylton's blog post is suggestive about the causes, but I don't really understand CPython's implementation well-enough to figure out how to get around this.

Here is an (overcomplicated?) example. If people have a simpler one that demos it, I'd like to hear it. The issue: the list comprehensions using next() are filled with the result from the last iteration.

edit: The Problem:

What exactly is going on with this, and how do I fix this? Do I have to use a standard for loop? Clearly the function is running the correct number of times, but the list comprehensions end up with the final value instead of the result of each loop.

Some hypotheses:

• generators?
• lazy filling of list comprehensions?

code

import itertools
def digit(n):
    digit_list = [ (x,False) for x in xrange(1,n+1)]
    digit_list[0] = (1,True)
    return itertools.cycle ( digit_list)

>>> D = digit(5)
>>> [D.next() for x in range(5)]
## This list comprehension works as expected
[(1, True), (2, False), (3, False), (4, False), (5, False)]

class counter(object):
    def __init__(self):
        self.counter = [ digit(4) for ii in range(2) ] 
        self.totalcount=0
        self.display = [0,] * 2
    def next(self):
        self.totalcount += 1
        self.display[-1] = self.counter[-1].next()[0]
        print self.totalcount, self.display
        return self.display

    def next2(self,*args):
        self._cycle(1)
        self.totalcount += 1
        print self.totalcount, self.display
        return self.display

    def _cycle(self,digit):
        d,first = self.counter[digit].next()
        #print digit, d, first
        #print self._display
        self.display[digit] = d
        if first and digit > 0:
            self._cycle(digit-1)


C = counter()
[C.next() for x in range(5)]
[C.next2() for x in range(5)]

OUTPUT

In [44]: [C.next() for x in range(6)]
1 [0, 1]
2 [0, 2]
3 [0, 3]
4 [0, 4]
5 [0, 1]
6 [0, 2]
Out[44]: [[0, 2], [0, 2], [0, 2], [0, 2], [0, 2], [0, 2]]

In [45]: [C.next2() for x in range(6)]
7 [0, 3]
8 [0, 4]
9 [1, 1]
10 [1, 2]
11 [1, 3]
12 [1, 4]
Out[45]: [[1, 4], [1, 4], [1, 4], [1, 4], [1, 4], [1, 4]]

# this should be:  [[0,3],[0,4]....[1,4]] or similar

Answer 1

The problem is that with return self.display you return a reference to this list (not a copy). So what you end up with is a list where each element is a reference to self.display. To illustrate, look at the following:

>>> a = [1,2]
>>> b = [a,a]
>>> b
[[1, 2], [1, 2]]
>>> a.append(3)
>>> b
[[1, 2, 3], [1, 2, 3]]

You probably want to use something like return self.display[:].