Python 3.x: Test if generator has elements remaining

Question

When I use a generator in a for loop, it seems to "know", when there are no more elements yielded. Now, I have to use a generator WITHOUT a for loop, and use next() by hand, to get the next element. My problem is, how do I know, if there are no more elements?

I know only: next() raises an exception (StopIteration), if there is nothing left, BUT isn't an exception a little bit too "heavy" for such a simple problem? Isn't there a method like has_next() or so?

The following lines should make clear, what I mean:

#!/usr/bin/python3  # define a list of some objects bar = ['abc', 123, None, True, 456.789]  # our primitive generator def foo(bar):     for b in bar:         yield b  # iterate, using the generator above print('--- TEST A (for loop) ---') for baz in foo(bar):     print(baz) print()  # assign a new iterator to a variable foobar = foo(bar)  print('--- TEST B (try-except) ---') while True:     try:         print(foobar.__next__())     except StopIteration:         break print()  # assign a new iterator to a variable foobar = foo(bar)  # display generator members print('--- GENERATOR MEMBERS ---') print(', '.join(dir(foobar))) 

The output is as follows:

--- TEST A (for loop) --- abc 123 None True 456.789  --- TEST B (try-except) --- abc 123 None True 456.789  --- GENERATOR MEMBERS --- __class__, __delattr__, __doc__, __eq__, __format__, __ge__, __getattribute__, __gt__, __hash__, __init__, __iter__, __le__, __lt__, __name__, __ne__, __new__, __next__, __reduce__, __reduce_ex__, __repr__, __setattr__, __sizeof__, __str__, __subclasshook__, close, gi_code, gi_frame, gi_running, send, throw 

Thanks to everybody, and have a nice day! :)

Answer 1

This is a great question. I'll try to show you how we can use Python's introspective abilities and open source to get an answer. We can use the dis module to peek behind the curtain and see how the CPython interpreter implements a for loop over an iterator.

>>> def for_loop(iterable): ...     for item in iterable: ...         pass  # do nothing ...      >>> import dis >>> dis.dis(for_loop)   2           0 SETUP_LOOP              14 (to 17)                3 LOAD_FAST                0 (iterable)                6 GET_ITER                      >>    7 FOR_ITER                 6 (to 16)               10 STORE_FAST               1 (item)     3          13 JUMP_ABSOLUTE            7          >>   16 POP_BLOCK                     >>   17 LOAD_CONST               0 (None)               20 RETURN_VALUE          

The juicy bit appears to be the FOR_ITER opcode. We can't dive any deeper using dis, so let's look up FOR_ITER in the CPython interpreter's source code. If you poke around, you'll find it in Python/ceval.c; you can view it here. Here's the whole thing:

    TARGET(FOR_ITER)         /* before: [iter]; after: [iter, iter()] *or* [] */         v = TOP();         x = (*v->ob_type->tp_iternext)(v);         if (x != NULL) {             PUSH(x);             PREDICT(STORE_FAST);             PREDICT(UNPACK_SEQUENCE);             DISPATCH();         }         if (PyErr_Occurred()) {             if (!PyErr_ExceptionMatches(                             PyExc_StopIteration))                 break;             PyErr_Clear();         }         /* iterator ended normally */         x = v = POP();         Py_DECREF(v);         JUMPBY(oparg);         DISPATCH(); 

Do you see how this works? We try to grab an item from the iterator; if we fail, we check what exception was raised. If it's StopIteration, we clear it and consider the iterator exhausted.

So how does a for loop "just know" when an iterator has been exhausted? Answer: it doesn't -- it has to try and grab an element. But why?

Part of the answer is simplicity. Part of the beauty of implementing iterators is that you only have to define one operation: grab the next element. But more importantly, it makes iterators lazy: they'll only produce the values that they absolutely have to.

Finally, if you are really missing this feature, it's trivial to implement it yourself. Here's an example:

class LookaheadIterator:      def __init__(self, iterable):         self.iterator = iter(iterable)         self.buffer = []      def __iter__(self):         return self      def __next__(self):         if self.buffer:             return self.buffer.pop()         else:             return next(self.iterator)      def has_next(self):         if self.buffer:             return True          try:             self.buffer = [next(self.iterator)]         except StopIteration:             return False         else:             return True   x  = LookaheadIterator(range(2))  print(x.has_next()) print(next(x)) print(x.has_next()) print(next(x)) print(x.has_next()) print(next(x))