Why does a class need __iter__() to return an iterator?

Question

Why does a class need to define __iter__() returning self, to get an iterator of the class?

class MyClass:
    def __init__(self):
        self.state = 0

    def __next__(self):
        self.state += 1
        if self.state > 4:
            raise StopIteration
        return self.state

myObj = MyClass()
for i in myObj:
    print(i)

Console log:

Traceback (most recent call last):
   for i in myObj:
TypeError: 'MyClass' object is not iterable

the answer https://stackoverflow.com/a/9884259/4515198, says

An iterator is an object with a next (Python 2) or __next__ (Python 3) method.

The task of adding the following:

def __iter__(self):
   return self

is to return an iterator, or an object of the class, which defines the __next__() method.

But, isn't the task of returning an object of MyClass (which defines the __next__() method) already done by the __new__() method, when MyClass is instantiated in the line myObj = MyClass() ?

Won't the objects of a class defining __next__() method, be iterators by themselves?

I have studied the questions What is the use of returning self in the __iter__ method? and Build a Basic Python Iterator, but I am still unable to understand the reason for having an __iter__() method returning self.

Answer 1

The answer to the question of why the __iter__() method is necessary is that for for-loops always start by calling iter() on an object to get an iterator. That is why even iterators themselved need an __iter__() method to work with for-loops. After for calls iter(), then it calls __next__() on the resulting iterator to obtain a value.

The rules for creating iterables and iterators are:

1) Iterables have an __iter__() method that returns an iterator.

2) Iterators have a __next__() method that returns a value, that updates the state, and that raises StopIteration when complete.

3) Iterators themselves have a __iter__() method that returns self. That means that all iterators are self-iterable.

The benefit of the last rule for iterators having an __iter__() method that returns self is that it allows us to pass around partially consumed iterators:

>>> s = 'hello world'
>>> it = iter(s)
>>> next(it)
'h'
>>> next(it)
'e'
>>> list(it)     # Doesn't start from the beginning
['l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd']

Here's another example that depends on iterators being self-iterable without restarting:

>>> s = 'hello world'
>>> it = iter(s)
>>> list(zip(it, it))
[('h', 'e'), ('l', 'l'), ('o', ' '), ('w', 'o'), ('r', 'l')]

Notes:

1) An alternative way to make an iterable is to supply a __getitem__() method that accepts consecutive indices and raises IndexError when complete. This is how str objects iterated in Python 2.

2) Some objects like files are their own iterator. That means that you can call next() directly on a file object. It also means that files cannot have multiple, independent iterators (the file object itself has the state tracking the position within the file).

3) The iterator design pattern described above isn't Python specific. It is a general purpose design pattern for many OOP languages: https://en.wikipedia.org/wiki/Iterator_pattern