Python asyncio ensure_future decorator

Question

Let's assume I'm new to asyncio. I'm using async/await to parallelize my current project, and I've found myself passing all of my coroutines to asyncio.ensure_future. Lots of stuff like this:

coroutine = my_async_fn(*args, **kwargs)
task = asyncio.ensure_future(coroutine)

What I'd really like is for a call to an async function to return an executing task instead of an idle coroutine. I created a decorator to accomplish what I'm trying to do.

def make_task(fn):
    def wrapper(*args, **kwargs):
        return asyncio.ensure_future(fn(*args, **kwargs))
    return wrapper

@make_task
async def my_async_func(*args, **kwargs):
    # usually making a request of some sort
    pass

Does asyncio have a built-in way of doing this I haven't been able to find? Am I using asyncio wrong if I'm lead to this problem to begin with?

Answer 1

asyncio had @task decorator in very early pre-released versions but we removed it.

The reason is that decorator has no knowledge what loop to use. asyncio don't instantiate a loop on import, moreover test suite usually creates a new loop per test for sake of test isolation.

Answer 2

Does asyncio have a built-in way of doing this I haven't been able to find?

No, asyncio doesn't have decorator to cast coroutine-functions into tasks.

Am I using asyncio wrong if I'm lead to this problem to begin with?

It's hard to say without seeing what you're doing, but I think it may happen to be true. While creating tasks is usual operation in asyncio programs I doubt you created this much coroutines that should be tasks always.

Awaiting for coroutine - is a way to "call some function asynchronously", but blocking current execution flow until it finished:

await some()

# you'll reach this line *only* when some() done 

Task on the other hand - is a way to "run function in background", it won't block current execution flow:

task = asyncio.ensure_future(some())

# you'll reach this line immediately

When we write asyncio programs we usually need first way since we usually need result of some operation before starting next one:

text = await request(url)

links = parse_links(text)  # we need to reach this line only when we got 'text'

Creating task on the other hand usually means that following further code doesn't depend of task's result. But again it doesn't happening always.

Since ensure_future returns immediately some people try to use it as a way to run some coroutines concurently:

# wrong way to run concurrently:
asyncio.ensure_future(request(url1))
asyncio.ensure_future(request(url2))
asyncio.ensure_future(request(url3))

Correct way to achieve this is to use asyncio.gather:

# correct way to run concurrently:
await asyncio.gather(
    request(url1),
    request(url2),
    request(url3),
)

May be this is what you want?

Upd:

I think using tasks in your case is a good idea. But I don't think you should use decorator: coroutine functionality (to make request) still is a separate part from it's concrete usage detail (it will be used as task). If requests synchronization controlling is separate from their's main functionalities it's also make sense to move synchronization into separate function. I would do something like this:

import asyncio


async def request(i):
    print(f'{i} started')
    await asyncio.sleep(i)
    print(f'{i} finished')
    return i


async def when_ready(conditions, coro_to_start):
    await asyncio.gather(*conditions, return_exceptions=True)
    return await coro_to_start


async def main():
    t = asyncio.ensure_future

    t1 = t(request(1))
    t2 = t(request(2))
    t3 = t(request(3))
    t4 = t(when_ready([t1, t2], request(4)))
    t5 = t(when_ready([t2, t3], request(5)))

    await asyncio.gather(t1, t2, t3, t4, t5)


if __name__ ==  '__main__':
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(main())
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())
        loop.close()