The Iterator Protocol. Is it Dark Magic?

Question

So I've been writing iterators for a while, and I thought that I understood them. But I've been struggling with some issues tonight, and the more I play with it, the more confused I become.

I thought that for an iterator you had to implement __iter__ and next (or __next__). And that when you first tried to iterate over the iterator the __iter__ method would be called, and then next would be called until an StopIteration was raised.

When I run this code though

class Iter(object):

    def __iter__(self):
        return iter([2, 4, 6])

    def next(self):
        for y in [1, 2, 3]:
            return y

iterable = Iter()
for x in iterable:
    print(x)

The output is 2 4 6. So __iter__ is being called, but not next. That does seem to match with the documentation that I found here. But then that raises a whole bunch more questions in my mind.

Specifically, what's the difference between a container type and iterator if it's not the implementation of next? How do I know before hand which way my class is going to be treated? And most importantly, if I want to write a class where my next method is called when I use for x in Iter(), how can I do that?

Answer 1

A list is iterable, but it is not an iterator. Compare and contrast:

>>> type([])
list
>>> type(iter([]))
list_iterator

Calling iter on a list creates and returns a new iterator object for iterating the contents of that list.

In your object, you just return a list iterator, specifically an iterator over the list [2, 4, 6], so that object knows nothing about yielding elements 1, 2, 3.

def __iter__(self):
    return iter([2, 4, 6])  # <-- you're returning the list iterator, not your own

Here's a more fundamental implementation conforming to the iterator protocol in Python 2, which doesn't confuse matters by relying on list iterators, generators, or anything fancy at all.

class Iter(object):

    def __iter__(self):
        self.val = 0
        return self

    def next(self):
        self.val += 1
        if self.val > 3:
            raise StopIteration
        return self.val