Python next() function fails while __next__ works on file?

Question

I was studying Mark Lutz's book on python called "Learning Python (5th Ed)". While perusing chapter 14 on Iteration and Comprehensions , I faced a problem in page 423.

There, we have created a file object returned by os.popen() call. It is an iterable object (has __iter__ method) but not an iterator (no __next__ method). So, this object should be called by neither next() built-in function nor __next__ method call. But somehow this object returns values on next dunder method call but raise exception on next() builtin calls.

In the book, the author said it is unusual. But he didn't provide any explanation for that.

Can anyone explain me how __next__ succeeds for that file-like object?

import os
file = os.popen("ls -l")
print(dir(file)) # this shows __iter__ but no __next__/ next
# print(next(file)) # this fails if uncommented, expected
print(file.__next__()) # this does NOT fail even there is no __next__

Similar code for a list that behaves as I expect:

file = [1]
print(dir(file)) # this shows __iter__ but no __next__/ next
# print(next(file)) # this fails if uncommented, expected
# print(file.__next__()) # this fails if uncommented, expected

Answer 1

os.popen returns a file-like proxy object, an instance of a _wrap_close helper class. This helper class implements __iter__, but it does not implement __next__. However, it does implement __getattr__, forwarding unknown attribute lookup to the underlying object it wraps:

def __getattr__(self, name):
    return getattr(self._stream, name)

When you call __next__ manually, __getattr__ forwards the lookup to the underlying object, the stdout stream of a subprocess.Popen instance, and this object has a __next__ method. However, next bypasses __getattr__. As a general principle, when Python needs to look up a special method to implement a core language feature or built-in function, it bypasses __getattr__, __getattribute__, and the instance dict, and directly searches the object's class and ancestor classes for the method.

There are a few exceptions to that principle, but it usually applies, and this is not one of the exceptions. Since the _wrap_close helper does not have an implementation of __next__, next raises an exception.