invoking yield for a generator in another function

Question

suppose I have some manager object. This object's API has a main_hook function, that gets another function f as it's argument, and runs the given f in a loop, doing some stuff in between each iteration:

def main_hook(self,f):
    while (self.shouldContinue()):
        #do some preparations
        f(self)
        #do some tear down

Now, I also have (more accurately, would like to have) a function stop_and_do_stuff, that once called, stops main_hook dead in it's tracks, returns the control to whichever func called main_hook, and after that func finished what's it doing, get control back to main_hook and continue. Basically the result will be the same as doing

def main_hook(self,f):
    while (self.shouldContinue()):
        #do some preparations
        yield
        #do some tear down

Except that instead yield I want to have a call to f(), while giving f the option to call self.stop_and_do_stuff()

I can't work around this by making f also a generator for 2 reasons:

1.f isn't part of my API - it's given to me by a user who uses my lib

2.Even if could ask him to use yield, the place in the code in which he will need to call stop_and_do_stuff won't be directly inside f, rather in some place in the function stack which will be inside f(), but not directly in it, e.g

def h(manager):
    #do stuff
    if should stop:
        manager.stop_and_do_stuff()
    #do more stuff
def g(manager):
    #some stuff
    if should stop:
        manager.stop_and_do_stuff()
    #more stuff
    if should stop again:
        manager.stop_and_do_stuff()  
    if should call h:
        h()
def f(manager):
    g(manager)

so if I choose to make f a generator, I also need to make g a generator and also h, otherwise this trick won't work.

Is there any solution to all of this? maybe I'm trying to solve it the wrong way?

(I know this question is long and ugly - it's the best I could do. If something isn't clear please tell me and I'll clarify it)

EDIT

Maybe pep 342 is the solution?

Answer 1

My previous answer describes how to do this in Python2, which is very ugly. But now I ran across PEP 380: Syntax for Delegating to a Subgenerator. That does exactly what you ask. The only problem is that it requires Python3. But that shouldn't really be a problem.

Here's how it works:

def worker():
    yield 1
    yield 2
    return 3

def main():
    yield 0
    value = yield from worker()
    print('returned %d' % value)
    yield 4

for m in main():
    print('generator yields %d' % m)

The result of this is:

generator yields 0
generator yields 1
generator yields 2
returned 3
generator yields 4

Exceptions are passed through the way you would expect.

Answer 2

I believe I should also add an answer from the other point of view, ie not trying to explain how you could achieve what we can understand of what you are trying to do, but why yield definitely couldn't possibly work.

When a function contains yield keyword it is deeply modified. It is still a callable but not a normal function any more : it becomes a factory that return an iterator.

From the caller's point of view there is no difference between the three implementations below (except that the yield one is so much simpler).

##########################################
print "Function iterator using yield",

def gen():
    for x in range(0, 10):
        yield x

f = gen()
try:
    while True:
        print f.next(),
except StopIteration:
    pass

for x in gen():
    print x,

print

#########################################
print "Class iterator defining iter and next",

class gen2(object):

    def __init__(self):
        self.index = 0;
        self.limit = 10;

    def __iter__(self):
        return self

    def next(self):
        if self.index >= self.limit:
            raise StopIteration
        self.index += 1;
        return self.index - 1;


f = gen2()
try:
    while True:
        print f.next(),
except StopIteration:
    pass

for x in gen2():
    print x,
print

#########################################
print "Function iterator using iter() and sentinel",
def gen3():
    def g3():
        if g3.index is None:
            g3.index = 0
        g3.index += 1;
        return g3.index - 1

    g3.index = None
    return iter(g3, 10)

f = gen3()
try:
    while True:
        print f.next(),
except StopIteration:
    pass

for x in gen3():
    print x,
print

Then you should understand that yield is not much about control flow, but about keeping call context inside variables. Once it is understood you have to decide if the API of main_loop really want to provide an iterator to it's caller. Then if so, if f may loop it must should also be an iterator (and there should be a loop around calls to f() like below).

def main_hook(self,f):
    while (self.shouldContinue()):
        #do some preparations
        for v in f(self):
            yield v
        #do some tear down

But you should not care if f() has to call inner functions g(), etc. That is completely irrelevant. You provide a lib and it is your user problem to call with an appropriate iterable. If you believe your lib user won't be able to, you will have to change the overall design.

Hope it helps.