Asyncio How do you use run_forever?

Question

What I want to do:

1. have an asyncio event loop that gets spun up
2. that loop is passed to various classes in my system for scheduling coroutines on
3. that loop is also used for handling the responses to events (ie, I have a Queue, some event handling code will place an item on that queue, and separate co-routines that await a get() on that queue to handle those values)
4. there is a main thread which "owns" the loop and is responsible for creating the loop, and at time of system shutdown will cancel any running tasks on the loop and close & stop the loop (cleanly shutdown)

My understanding is because of #3, something needs to call run_forever() on the loop to ensure that tasks get scheduled on the loop. But if I call run_forever() then my main thread blocks, never to terminate.

What I've tried:

Spawn a thread, passing in the loop, and then call run_forever in the thread. This means though that my unit tests never finish. The gist:

def __start_background_loop(loop):
    def run_forever(loop):
        loop.run_forever()

    # because run_forever() will block the current thread, we spawn
    # a subthread to issue that call in.
    thread = Thread(target=run_forever, args=(loop,))
    thread.start()

def __end_background_loop(loop):
    for task in Task.all_tasks(loop):
        task.cancel()
    loop.stop()

Answer 1

There are two possible approaches: you can run the event loop in the main thread or in a background thread. If you run it in the main thread, you need to run_forever (or run_until_complete(main()) or equivalent) as the very last step of the program initialization. In that case the main thread will "block", but that's ok because its event loop will be live and respond to outside events, allowing the program to function. A single "blocking" call to the event loop that dispatches coroutines and callbacks is how asyncio is designed to be run.

In cases where this is impractical, such as programs that contain a large body of synchronous code, or those that already communicate between several threads, it is often a better idea to create a dedicated thread and run the event loop in it. In that case you must be very careful not to communicate with the event loop other than with calls to loop.call_soon_threadsafe() and asyncio.run_coroutine_threadsafe(). For example, __end_background_loop must be invoked using loop.call_soon_threadsafe(__end_background_loop) because it interacts with the tasks and the event loop. This applies to all interactions with the event loop - for example, calling loop.stop() from another thread is not allowed, it must be spelled as loop.call_soon_threadsafe(loop.stop). Of course, calling loop functions from asyncio callbacks and coroutines is fine because those will always be run in the same thread the event loop runs in.