Difference between `yield from foo()` and `for x in foo(): yield x`

Question

In Python most examples of yield from explain it with saying that

yield from foo()

is similar to

for x in foo(): yield x

On the other hand it doesn't seem to be exactly the same and there's some magic thrown in. I feel a bit uneasy about using a function that does magic that I don't understand. What do I have to know about the magic of yield from to avoid getting into a situation where the magic does something I don't expect? What advantages does the magic provide, that I should be aware of?

Answer 1

When foo() returns a regular iterable, the two are equivalent. The 'magic' comes into play when foo() is a generator too. At that moment, the yield from foo() and for x in foo(): yield x cases differ materially.

A generator can be sent data too, using the generator.send() method. When you use the for loop, the yield x expression 'receives' the sent data; the foo() generator will never see this. But when you use yield from the sent data goes straight to whatever yield expression the delegated-to generator is currently paused at. In other words, yield from passes on the sent data so the delegated-to generator can receive it instead.

You can also raise exceptions in a generator, with generator.throw(); with the for loop case, the exception is raised from the yield x line, while with yield from the exception is passed on again; the exception is raised inside foo() instead.

Together, this means that yield from in essence replaces the current generator for the duration of the delegated iteration.

The delegated-to generator also gets to communicate with the parent generator, when done the .value attribute of the StopIteration exception raised is returned as the value of the yield from expression. You can set the value of that exception by using return <expression> in the delegated-to foo() generator, or you can use raise StopIteration(<expression>) explicitly.

yield from was introduced into the language with PEP 380: Syntax for Delegating to a Subgenerator.