How to raise exceptions in python asyncio background task

Question

Problem

I have a few tasks that run continuously, but one of them occasionally needs to restart so this one is run in the background. How can exceptions from this background task be raised immediately? In the following example, the exception is not raised until the next attempt to restart, which can be very infrequent in real applications, so can undesirably go unnoticed for a long time.

Example

https://replit.com/@PatrickPei/how-to-raise-exceptions-in-python-asyncio-background-task

This example runs 3 tasks:

1. foo, which raises no exceptions
2. bar, which raises an exception after 6 iterations
3. on_interval, which restarts bar every 5 seconds

import asyncio

task = None
i = 0


async def foo():
    while True:
        print("foo")
        await asyncio.sleep(1)


async def bar():
    while True:
        global i
        i += 1
        if i > 4:
            raise ValueError()

        print("bar", i)
        await asyncio.sleep(1)


async def on_interval(n):
    while True:
        await asyncio.sleep(n)

        # Cancel task.
        global task
        print("Canceling bar")
        task.cancel()
        try:
            await task
        except asyncio.CancelledError:
            pass

        # Restart task.
        print("Restarting bar")
        task = asyncio.create_task(bar())


async def main():
    # Start background task.
    print("Starting bar")
    global task
    task = asyncio.create_task(bar())

    # Start other tasks.
    await asyncio.gather(
        foo(),
        on_interval(3),
    )


if __name__ == "__main__":
    asyncio.run(main())

Output

bar iterates 4 times and raises an exception, which is not caught until the next restart, as shown by 3 foo iterations after bar 4. This is a problem when there is a lot of time in between restarts since exceptions go unnoticed for a long time.

Starting bar
bar 1
foo
bar 2
foo
bar 3
foo
Canceling bar
Restarting bar
bar 4
foo
foo
foo
Canceling bar
Traceback (most recent call last):
  File "~/example.py", line 60, in <module>
    asyncio.run(main())
  File "~/.pyenv/versions/3.10.5/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "~/.pyenv/versions/3.10.5/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "~/example.py", line 53, in main
    await asyncio.gather(
  File "~/example.py", line 37, in on_interval
    await task
  File "~/example.py", line 22, in bar
    raise ValueError()
ValueError

Attempts

• Started another task to check asyncio.Task.exception, but this is cumbersome because every background task needs another busy loop to help raise its exceptions.
• Tried asyncio.Task.add_done_callback but since the background task is still not awaited until the next restart, it only logs the error and does not stop the other task foo.

Answer 1

In python 3.11, using async with the asynchronous context manager and asyncio.TaskGroup() solves this problem simply.

import asyncio

i = 0


async def foo():
    while True:
        print("foo")
        await asyncio.sleep(1)


async def bar():
    while True:
        global i
        i += 1
        if i > 14:
            raise ValueError()

        print("bar", i)
        await asyncio.sleep(1)


async def on_interval(n):
    while True:
        async with asyncio.TaskGroup() as tg1:
            print("Restarting bar")
            task2 = tg1.create_task(bar())

            await asyncio.sleep(n)

            print("Canceling bar")
            task2.cancel()
         


async def main():
    async with asyncio.TaskGroup() as tg:
        task1 = tg.create_task(foo())
        task3 = tg.create_task(on_interval(3))


asyncio.run(main())

-------------------------

foo
Restarting bar
bar 1
foo
bar 2
foo
bar 3
Canceling bar
foo
Restarting bar
bar 4
  + Exception Group Traceback (most recent call last):
  |   File "C:\Users\ф\PycharmProjects\tkinter\rrr.py", line 42, in <module>
  |     asyncio.run(main())
  |   File "C:\Users\ф\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 190, in run
  |     return runner.run(main)
  |            ^^^^^^^^^^^^^^^^
  |   File "C:\Users\ф\AppData\Local\Programs\Python\Python311\Lib\asyncio\runners.py", line 118, in run
  |     return self._loop.run_until_complete(task)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "C:\Users\ф\AppData\Local\Programs\Python\Python311\Lib\asyncio\base_events.py", line 650, in run_until_complete
  |     return future.result()
  |            ^^^^^^^^^^^^^^^
  |   File "C:\Users\ф\PycharmProjects\tkinter\rrr.py", line 37, in main
  |     async with asyncio.TaskGroup() as tg:
  |   File "C:\Users\ф\AppData\Local\Programs\Python\Python311\Lib\asyncio\taskgroups.py", line 135, in __aexit__
  |     raise me from None
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Exception Group Traceback (most recent call last):
    |   File "C:\Users\ф\PycharmProjects\tkinter\rrr.py", line 25, in on_interval
    |     async with asyncio.TaskGroup() as tg1:
    |   File "C:\Users\ф\AppData\Local\Programs\Python\Python311\Lib\asyncio\taskgroups.py", line 135, in __aexit__
    |     raise me from None
    | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
    +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      |   File "C:\Users\ф\PycharmProjects\tkinter\rrr.py", line 17, in bar
      |     raise ValueError()
      | ValueError
      +------------------------------------
foo

Process finished with exit code 1