Python 3.5 async/await with real code example

Question

I've read tons of articles and tutorial about Python's 3.5 async/await thing. I have to say I'm pretty confused, because some use get_event_loop() and run_until_complete(), some use ensure_future(), some use asyncio.wait(), and some use call_soon().

It seems like I have a lot choices, but I have no idea if they are completely identical or there are cases where you use loops and there are cases where you use wait().

But the thing is all examples work with asyncio.sleep() as simulation of real slow operation which returns an awaitable object. Once I try to swap this line for some real code the whole thing fails. What the heck are the differences between approaches written above and how should I run a third-party library which is not ready for async/await. I do use the Quandl service to fetch some stock data.

 import asyncio  import quandl   async def slow_operation(n):      # await asyncio.sleep(1) # Works because it's await ready.      await quandl.Dataset(n) # Doesn't work because it's not await ready.    async def main():      await asyncio.wait([          slow_operation("SIX/US9884981013EUR4"),          slow_operation("SIX/US88160R1014EUR4"),      ])   # You don't have to use any code for 50 requests/day.  quandl.ApiConfig.api_key = "MY_SECRET_CODE"   loop = asyncio.get_event_loop()  loop.run_until_complete(main()) 

I hope you get the point how lost I feel and how simple thing I would like to have running in parallel.

Answer 1

If a third-party library is not compatible with async/await then obviously you can't use it easily. There are two cases:

1. Let's say that the function in the library is asynchronous and it gives you a callback, e.g.

   def fn(..., clb):     ... 

   So you can do:

   def on_result(...):     ...  fn(..., on_result) 

   In that case you can wrap such functions into the asyncio protocol like this:

   from asyncio import Future  def wrapper(...):     future = Future()     def my_clb(...):         future.set_result(xyz)     fn(..., my_clb)     return future 

   (use future.set_exception(exc) on exception)

   Then you can simply call that wrapper in some async function with await:

   value = await wrapper(...) 

   Note that await works with any Future object. You don't have to declare wrapper as async.

2. If the function in the library is synchronous then you can run it in a separate thread (probably you would use some thread pool for that). The whole code may look like this:

   import asyncio import time from concurrent.futures import ThreadPoolExecutor  # Initialize 10 threads THREAD_POOL = ThreadPoolExecutor(10)  def synchronous_handler(param1, ...):     # Do something synchronous     time.sleep(2)     return "foo"  # Somewhere else async def main():     loop = asyncio.get_event_loop()     futures = [         loop.run_in_executor(THREAD_POOL, synchronous_handler, param1, ...),         loop.run_in_executor(THREAD_POOL, synchronous_handler, param1, ...),         loop.run_in_executor(THREAD_POOL, synchronous_handler, param1, ...),     ]     await asyncio.wait(futures)     for future in futures:         print(future.result())  with THREAD_POOL:     loop = asyncio.get_event_loop()     loop.run_until_complete(main()) 

If you can't use threads for whatever reason then using such a library simply makes entire asynchronous code pointless.

Note however that using synchronous library with async is probably a bad idea. You won't get much and yet you complicate the code a lot.

Answer 2

You can take a look at the following simple working example from here. By the way it returns a string worth reading :-)

import aiohttp import asyncio  async def fetch(client):   async with client.get('https://docs.aiohttp.org/en/stable/client_reference.html') as resp:     assert resp.status == 200     return await resp.text()  async def main():   async with aiohttp.ClientSession() as client:     html = await fetch(client)     print(html)  loop = asyncio.get_event_loop() loop.run_until_complete(main())