Attempting to understand yield as an expression

Question

I'm playing around with generators and generator expressions and I'm not completely sure that I understand how they work (some reference material):

>>> a = (x for x in range(10))
>>> next(a)
0
>>> next(a)
1
>>> a.send(-1)
2
>>> next(a)
3

So it looks like generator.send was ignored. That makes sense (I guess) because there is no explicit yield expression to catch the sent information ...

However,

>>> a = ((yield x) for x in range(10))
>>> next(a)
0
>>> print next(a)
None
>>> print next(a)
1
>>> print next(a)
None
>>> a.send(-1)  #this send is ignored, Why? ... there's a yield to catch it...
2
>>> print next(a)
None
>>> print next(a)
3
>>> a.send(-1)  #this send isn't ignored
-1

I understand this is pretty far out there, and I (currently) can't think of a use-case for this (so don't ask;)

I'm mostly just exploring to try to figure out how these various generator methods work (and how generator expressions work in general). Why does my second example alternate between yielding a sensible value and None? Also, Can anyone explain why one of my generator.send's was ignored while the other wasn't?

Answer 1

The confusion here is that the generator expression is doing a hidden yield. Here it is in function form:

def foo():
    for x in range(10):
        yield (yield x)

When you do a .send(), what happens is the inner yield x gets executed, which yields x. Then the expression evaluates to the value of the .send, and the next yield yields that. Here it is in clearer form:

def foo():
    for x in range(10):
        sent_value = (yield x)
        yield sent_value

Thus the output is very predictable:

>>> a = foo()
#start it off
>>> a.next() 
0
#execution has now paused at "sent_value = ?"
#now we fill in the "?". whatever we send here will be immediately yielded.
>>> a.send("yieldnow") 
'yieldnow'
#execution is now paused at the 'yield sent_value' expression
#as this is not assigned to anything, whatever is sent now will be lost
>>> a.send("this is lost") 
1
#now we're back where we were at the 'yieldnow' point of the code
>>> a.send("yieldnow") 
'yieldnow'
#etc, the loop continues
>>> a.send("this is lost")
2
>>> a.send("yieldnow")
'yieldnow'
>>> a.send("this is lost")
3
>>> a.send("yieldnow")
'yieldnow'

EDIT: Example usage. By far the coolest one I've seen so far is twisted's inlineCallbacks function. See here for an article explaining it. The nub of it is it lets you yield functions to be run in threads, and once the functions are done, twisted sends the result of the function back into your code. Thus you can write code that heavily relies on threads in a very linear and intuitive manner, instead of having to write tons of little functions all over the place.

See the PEP 342 for more info on the rationale of having .send work with potential use cases (the twisted example I provided is an example of the boon to asynchronous I/O this change offered).

Answer 2

You're confusing yourself a bit because you actually are generating from two sources: the generator expression (... for x in range(10)) is one generator, but you create another source with the yield. You can see that if do list(a) you'll get [0, None, 1, None, 2, None, 3, None, 4, None, 5, None, 6, None, 7, None, 8, None, 9, None].

Your code is equivalent to this:

>>> def gen():
...     for x in range(10):
...         yield (yield x)

Only the inner yield ("yield x") is "used" in the generator --- it is used as the value of the outer yield. So this generator iterates back and forth between yielding values of the range, and yielding whatever is "sent" to those yields. If you send something to the inner yield, you get it back, but if you happen to send on an even-numbered iteration, the send is sent to the outer yield and is ignored.

Answer 3

This generator translates into:

for i in xrange(10):
    x = (yield i)
    yield x

Result of second call to send()/next() are ignored, because you do nothing with result of one of yields.