Python : Behaviour of send() in generators

Question

I was experimenting with generators in python 3 and wrote this rather contrived generator :

def send_gen():
    print("    send_gen(): will yield 1")
    x = yield 1
    print("    send_gen(): sent in '{}'".format(x))
    # yield  # causes StopIteration when left out


gen = send_gen()
print("yielded {}".format(gen.__next__()))

print("running gen.send()")
gen.send("a string")

Output:

    send_gen(): will yield 1
yielded 1
running gen.send()
    send_gen(): sent in 'a string'
Traceback (most recent call last):
  File "gen_test.py", line 12, in <module>
    gen.send("a string")
StopIteration

So gen.__next__() reaches the line x = yield 1 and yields 1. I thought x would be assigned to None, then gen.send() would look for the next yield statement because x = yield 1 is "used", then get a StopIteration.

Instead, what seems to have happened is that x gets sent "a string", which is printed, then then python attempts to look for the next yield and gets a StopIteration.

So i try this:

def send_gen():
    x = yield 1
    print("    send_gen(): sent in '{}'".format(x))


gen = send_gen()
print("yielded : {}".format(gen.send(None)))

Output :

yielded : 1

But now there's no error. send() doesn't appear to have tried to look for the next yield statement after assigning x to None.

Why is the behaviour slightly different ? Does this have to do with how I started the generators ?

Answer 1

The behaviour is not different; you never advanced beyond the first yield expression in the generator in the second setup. Note that StopIteration is not an error; it is normal behaviour, the expected signal to be fired whenever a generator has ended. In your second example, you just never reached the end of the generator.

Whenever a generator reaches a yield expression, execution pauses right there, the expression can't produce anything inside the generator until it is resumed. Either gen.__next__() or a gen.send() will both resume execution from that point, with the yield expression either producing the value passed in by gen.send(), or None. You could see gen.__next__() as a gen.send(None) if that helps. The one thing to realise here is that gen.send() has yield return the sent value first, and then the generator continues on to the next yield.

So, given your first example generator, this happens:

1. gen = send_gen() creates the generator object. The code is paused at the very top of the function, nothing is executed.

2. You either call gen.__next__() or gen.send(None); the generator commences and executes until the first yield expression:

   print("    send_gen(): will yield 1")
   yield 1
   

   and execution now pauses. The gen.__next__() or gen.send(None) calls now return 1, the value yielded by yield 1. Because the generator is now paused, the x = ... assignment can't yet take place! That'll only happen when the generator is resumed again.

3. You call gen.send("a string") in your first example, don't make any call in the second. So for the first example, the generator function is resumed now:

   x = <return value of the yield expression>  # 'a string' in this case
   print("    send_gen(): sent in '{}'".format(x))
   

   and now the function ends, so StopIteration is raised.

Because you never resumed the generator in your second example, the end of the generator is not reached and no StopIteration exception is raised.

Note that because a generator starts at the top of a function, there is no yield expression at that point to return whatever you sent with gen.send() so a first gen.send() value must always be None or an exception is raised. It is best to use an explicit gen.__next__() (or, rather a next(gen) function call) to 'prime' the generator so it'll be paused at the first yield expression.