How to use asyncio event loop in library function

Question

I'm trying to create a function performing some asynchronous operations using asyncio, users of this function should not need to know that asyncio is involved under the hood. I'm having a very hard time understanding how this shall be done with the asyncio API as most functions seem to operate under some global loop-variable accessed with get_event_loop and calls to this are affected by the global state inside this loop.

I have four examples here where two (foo1 and foo3) seem to be reasonable use cases but they all show very strange behaviors:

async def bar(loop):
    # Disregard how simple this is, it's just for example
    s = await asyncio.create_subprocess_exec("ls", loop=loop)


def foo1():
    # Example1: Just use get_event_loop
    loop = asyncio.get_event_loop()
    loop.run_until_complete(asyncio.wait_for(bar(loop), 1000))
    # On exit this is written to stderr:
    #    Exception ignored in: <bound method BaseEventLoop.__del__ of <_UnixSelectorEventLoop running=False closed=True debug=False>>
    #    Traceback (most recent call last):
    #      File "/usr/lib/python3.5/asyncio/base_events.py", line 510, in __del__
    #      File "/usr/lib/python3.5/asyncio/unix_events.py", line 65, in close
    #      File "/usr/lib/python3.5/asyncio/unix_events.py", line 146, in remove_signal_handler
    #      File "/usr/lib/python3.5/signal.py", line 47, in signal
    #    TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object


def foo2():
    # Example2: Use get_event_loop and close it when done
    loop = asyncio.get_event_loop()
    loop.run_until_complete(asyncio.wait_for(bar(loop), 1000))  # RuntimeError: Event loop is closed  --- if foo2() is called twice
    loop.close()


def foo3():
    # Example3: Always use new_event_loop
    loop = asyncio.new_event_loop()
    loop.run_until_complete(asyncio.wait_for(bar(loop), 1000)) #RuntimeError: Cannot add child handler, the child watcher does not have a loop attached
    loop.close()


def foo4():
    # Example4: Same as foo3 but also set_event_loop to the newly created one
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)        # Polutes global event loop, callers of foo4 does not expect this.
    loop.run_until_complete(asyncio.wait_for(bar(loop), 1000))  # OK
    loop.close()

None of these functions work and i don't see any other obvious way to do it, how is asyncio supposed to be used? It's seems like it's only designed to be used under the assumption that the entry point of the application is the only place where you can create and close the global loop. Do i have to fiddle around with event loop policies?

foo3 seems like the correct solution but i get an error even though i explicitly pass along loop, because deep down inside create_subprocess_exec it is using the current policy to get a new loop which is None, is this a bug in asyncio subprocess?

I'm using Python 3.5.3 on Ubuntu.

Answer 1

foo1 error happens because you didn't close event loop, see this issue.

foo2 because you can't reuse closed event loop.

foo3 because you didn't set new event loop as global.

foo4 is almost what you want, all you left to do is store old event loop and set it back as global after bar executed:

import asyncio


async def bar():
    # After you set new event loop global,
    # there's no need to pass loop as param to bar or anywhere else.
    process = await asyncio.create_subprocess_exec("ls")
    await process.communicate()


def sync_exec(coro):  # foo5
    old_loop = asyncio.get_event_loop()
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    try:
        loop.run_until_complete(coro)
    finally:
        loop.close()
        asyncio.set_event_loop(old_loop)


sync_exec(asyncio.wait_for(bar(), 1000))

One more important thing: it's not clear why you want to hide using of asyncio behind some sync functions, but usually it's bad idea. Whole thing about one global event loop is to allow user to run different concurrent jobs in this single event loop. You're trying to take away this possibility. I think you should reconsider this decision.

Answer 2

Upgrade to Python 3.6, then foo1() will work, without the need to explicitly close the default event loop.

Not the answer i was hoping for as we only use 3.5 :(