Python asynchronous callbacks and generators

Question

I'm trying to convert a synchronous library to use an internal asynchronous IO framework. I have several methods that look like this:

def foo:
  ....
  sync_call_1()   # synchronous blocking call
  ....
  sync_call_2()   # synchronous blocking call
  ....
  return bar

For each of the synchronous functions (sync_call_*), I have written a corresponding async function that takes a a callback. E.g.

def async_call_1(callback=none):
  # do the I/O
  callback()

Now for the python newbie question -- whats the easiest way to translate the existing methods to use these new async methods instead? That is, the method foo() above needs to now be:

def async_foo(callback):
  # Do the foo() stuff using async_call_*
  callback()

One obvious choice is to pass a callback into each async method which effectively "resumes" the calling "foo" function, and then call the callback global at the very end of the method. However, that makes the code brittle, ugly and I would need to add a new callback for every call to an async_call_* method.

Is there an easy way to do that using a python idiom, such as a generator or coroutine?

Answer 1

UPDATE: take this with a grain of salt, as I'm out of touch with modern python async developments, including gevent and asyncio and don't actually have serious experience with async code.

---

There are 3 common approaches to thread-less async coding in Python:

1. Callbacks - ugly but workable, Twisted does this well.

2. Generators - nice but require all your code to follow the style.

3. Use Python implementation with real tasklets - Stackless (RIP) and greenlet.

Unfortunately, ideally the whole program should use one style, or things become complicated. If you are OK with your library exposing a fully synchronous interface, you are probably OK, but if you want several calls to your library to work in parallel, especially in parallel with other async code, then you need a common event "reactor" that can work with all the code.

So if you have (or expect the user to have) other async code in the application, adopting the same model is probably smart.

If you don't want to understand the whole mess, consider using bad old threads. They are also ugly, but work with everything else.

If you do want to understand how coroutines might help you - and how they might complicate you, David Beazley's "A Curious Course on Coroutines and Concurrency" is good stuff.

Greenlets might be actualy the cleanest way if you can use the extension. I don't have any experience with them, so can't say much.

Answer 2

There are several way for multiplexing tasks. We can't say what is the best for your case without deeper knowledge on what you are doing. Probably the most easiest/universal way is to use threads. Take a look at this question for some ideas.