Run tasks asynchrounous with Python 3.6 asyncio

Question

I started working on a new Python 3.6 project that communicates with AWS Boto. Since spinning up EC2 instances takes some time, I started working with the asyncio library, I am having some troubles however understanding it.

I want to asynchronously spin up 2 EC2 instances. However, if I call run_tests I get the following error:

ERROR:asyncio:Task was destroyed but it is pending!

This is currently my code:

from manager import Manager
import asyncio


async def run_new_vm(manager, loop, vm_name):
    new_instance = manager.launch_ec2_instance(vm_name)
    task = loop.create_task(new_instance)
    task.add_done_callback(lambda f: do_something(manager, f.result()))


def do_something(manager, instance):
    // Do stuff once the instance is usable


async def one_service_per_vm(n, manager, loop):
    for x in range (0, n):
        print('Started with number %s.' % x)
        loop.create_task(run_new_vm(manager, loop, n))


def run_tests():
    loop = asyncio.get_event_loop()
    m = Manager()
    loop.run_until_complete(one_service_per_vm(2, m, loop))
    loop.close()

What am I doing wrong?

Answer 1

You're using create_task to schedule jobs on the loop, but nothing waits for them to complete. The one_service_per_vm will return immediately.

You can await for many tasks with asyncio.gather

# ...

async def one_service_per_vm(n, manager, loop):
    tasks = [run_new_vm(manager, loop, n) for x in range (0, n)]
    await asyncio.gather(*tasks, loop=loop)


def run_tests():
    loop = asyncio.get_event_loop()
    m = Manager()
    loop.run_until_complete(one_service_per_vm(2, m, loop))
    loop.close()