Python asyncio task list generation without executing the function

Question

While working in asyncio, I'm trying to use a list comprehension to build my task list. The basic form of the function is as follows:

import asyncio
import urllib.request as req
@asyncio.coroutine
def coro(term):
    print(term)
    google = "https://www.google.com/search?q=" + term.replace(" ", "+") + "&num=100&start=0"
    request = req.Request(google, None, headers) 
    (some beautiful soup stuff)

My goal is to use a list of terms to create my task list:

terms = ["pie", "chicken" ,"things" ,"stuff"]    
tasks=[
    coro("pie"),
    coro("chicken"),
    coro("things"),
    coro("stuff")]

My initial thought was:

loop = asyncio.get_event_loop()
tasks = [my_coroutine(term) for term in terms]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

This doesn't create the task list it runs the function during the list comprehension. Is there a way to use a shortcut to create the task list wihout writing every task?

Answer 1

Your HTTP client does not support asyncio, and you will not get the expected results. Try this to see .wait() does work as you expected:

import asyncio
import random

@asyncio.coroutine
def my_coroutine(term):
    print("start", term)
    yield from asyncio.sleep(random.uniform(1, 3))
    print("end", term)


terms = ["pie", "chicken", "things", "stuff"]
loop = asyncio.get_event_loop()
tasks = [my_coroutine(term) for term in terms]
print("Here we go!")
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

If you use asyncio.gather() you get one future encapsulating all your tasks, which can be easily canceled with .cancel(), here demonstrated with python 3.5+ async def/await syntax (but works the same with @coroutine and yield from):

import asyncio

import random


async def my_coroutine(term):
    print("start", term)
    n = random.uniform(0.2, 1.5)
    await asyncio.sleep(n)
    print("end", term)
    return "Term {} slept for {:.2f} seconds".format(term, n)


async def stop_all():
    """Cancels all still running tasks after one second"""
    await asyncio.sleep(1)
    print("stopping")
    fut.cancel()
    return ":-)"


loop = asyncio.get_event_loop()
terms = ["pie", "chicken", "things", "stuff"]
tasks = (my_coroutine(term) for term in terms)
fut = asyncio.gather(stop_all(), *tasks, return_exceptions=True)

print("Here we go!")
loop.run_until_complete(fut)

for task_result in fut.result():
    if not isinstance(task_result, Exception):
        print("OK", task_result)
    else:
        print("Failed", task_result)

loop.close()

And finally, if you want to use an async HTTP client, try aiohttp. First install it with:

pip install aiohttp

then try this example, which uses asyncio.as_completed:

import asyncio

import aiohttp


async def fetch(session, url):
    print("Getting {}...".format(url))
    async with session.get(url) as resp:
        text = await resp.text()
    return "{}: Got {} bytes".format(url, len(text))


async def fetch_all():
    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, "http://httpbin.org/delay/{}".format(delay))
                 for delay in (1, 1, 2, 3, 3)]
        for task in asyncio.as_completed(tasks):
            print(await task)
    return "Done."


loop = asyncio.get_event_loop()
resp = loop.run_until_complete(fetch_all())
print(resp)
loop.close()