Python generator pre-fetch?

Question

I have a generator that takes a long time for each iteration to run. Is there a standard way to have it yield a value, then generate the next value while waiting to be called again?

The generator would be called each time a button is pressed in a gui and the user would be expected to consider the result after each button press.

EDIT: a workaround might be:

def initialize():
    res = next.gen()

def btn_callback()
    display(res)
    res = next.gen()
    if not res:
       return

Answer 1

If I wanted to do something like your workaround, I'd write a class like this:

class PrefetchedGenerator(object):
    def __init__(self, generator):
         self._data = generator.next()
         self._generator = generator
         self._ready = True

    def next(self):
        if not self._ready:
            self.prefetch()
        self._ready = False
        return self._data

    def prefetch(self):
        if not self._ready:
            self._data = self._generator.next()
            self._ready = True

It is more complicated than your version, because I made it so that it handles not calling prefetch or calling prefetch too many times. The basic idea is that you call .next() when you want the next item. You call prefetch when you have "time" to kill.

Your other option is a thread..

class BackgroundGenerator(threading.Thread):
    def __init__(self, generator):
        threading.Thread.__init__(self)
        self.queue = Queue.Queue(1)
        self.generator = generator
        self.daemon = True
        self.start()

    def run(self):
        for item in self.generator:
            self.queue.put(item)
        self.queue.put(None)

    def next(self):
            next_item = self.queue.get()
            if next_item is None:
                 raise StopIteration
            return next_item

This will run separately from your main application. Your GUI should remain responsive no matter how long it takes to fetch each iteration.

Answer 2

No. A generator is not asynchronous. This isn't multiprocessing.

If you want to avoid waiting for the calculation, you should use the multiprocessing package so that an independent process can do your expensive calculation.

You want a separate process which is calculating and enqueueing results.

Your "generator" can then simply dequeue the available results.