next() doesn't play nice with any/all in python

Question

I ran down a bug today that came about because I was using next() to extract a value, and 'not found' emits a StopIteration.

Normally that would halt the program, but the function using next was being called inside an all() iteration, so the all just terminated early and returned True.

Is this an expected behavior? Are there style guides that help avoid this kind of thing?

Simplified example:

def error(): return next(i for i in range(3) if i==10) error() # fails with StopIteration all(error() for i in range(2)) # returns True 

Answer 1

While this is the default behaviour in Python versions up to and including 3.6, it's considered to be a mistake in the language, and is scheduled to change in Python 3.7 so that an exception is raised instead.

As PEP 479 says:

The interaction of generators and StopIteration is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, StopIteration raised accidentally inside a generator function will be interpreted as the end of the iteration by the loop construct driving the generator.

From Python 3.5 onwards, it's possible to change the default behaviour to that scheduled for 3.7. This code:

# gs_exc.py  from __future__ import generator_stop  def error():     return next(i for i in range(3) if i==10)  all(error() for i in range(2)) 

… raises the following exception:

Traceback (most recent call last):   File "gs_exc.py", line 8, in <genexpr>     all(error() for i in range(2))   File "gs_exc.py", line 6, in error     return next(i for i in range(3) if i==10) StopIteration  The above exception was the direct cause of the following exception:  Traceback (most recent call last):   File "gs_exc.py", line 8, in <module>     all(error() for i in range(2)) RuntimeError: generator raised StopIteration 

In Python 3.5 and 3.6 without the __future__ import, a warning is raised. For example:

# gs_warn.py  def error():     return next(i for i in range(3) if i==10)  all(error() for i in range(2)) 

$ python3.5 -Wd gs_warn.py  gs_warn.py:6: PendingDeprecationWarning: generator '<genexpr>' raised StopIteration   all(error() for i in range(2)) 

$ python3.6 -Wd gs_warn.py  gs_warn.py:6: DeprecationWarning: generator '<genexpr>' raised StopIteration   all(error() for i in range(2)) 

Answer 2

The problem isn't in using all, it's that you have a generator expression as the parameter to all. The StopIteration gets propagated to the generator expression, which doesn't really know where it originated, so it does the usual thing and ends the iteration.

You can see this by replacing your error function with something that raises the error directly:

def error2(): raise StopIteration  >>> all(error2() for i in range(2)) True 

The final piece of the puzzle is knowing what all does with an empty sequence:

>>> all([]) True 

If you're going to use next directly, you should be prepared to catch StopIteration yourself.

Edit: Nice to see that the Python developers consider this a bug and are taking steps to change it in 3.7.