python asyncio - how to wait for a cancelled shielded task?

Question

If I have a coroutine which runs a task which should not be cancelled, I will wrap that task in asyncio.shield().

It seems the behavior of cancel and shield is not what I would expect. If I have a task wrapped in shield and I cancel it, the await-ing coroutine returns from that await statement immediately rather than awaiting for the task to finish as shield would suggest. Additionally, the task that was run with shield continues to run but its future is now cancelled an not await-able.

From the docs:

except that if the coroutine containing it is cancelled, the Task running in something() is not cancelled. From the point of view of something(), the cancellation did not happen. Although its caller is still cancelled, so the “await” expression still raises a CancelledError.

These docs do not imply strongly that the caller is cancelled potentially before the callee finishes, which is the heart of my issue.

What is the proper method to shield a task from cancellation and then wait for it to complete before returning.

It would make more sense if asyncio.shield() raised the asyncio.CancelledError after the await-ed task has completed, but obviously there is some other idea going on here that I don't understand.

Here is a simple example:

import asyncio

async def count(n):
  for i in range(n):
    print(i)
    await asyncio.sleep(1)

async def t():
  try:
    await asyncio.shield(count(5))
  except asyncio.CancelledError:
    print('This gets called at 3, not 5')

  return 42

async def c(ft):
  await asyncio.sleep(3)

  ft.cancel()

async def m():
  ft = asyncio.ensure_future(t())
  ct = asyncio.ensure_future(c(ft))

  r = await ft

  print(r)

loop = asyncio.get_event_loop()
loop.run_until_complete(m())

# Running loop forever continues to run shielded task
# but I'd rather not do that
#loop.run_forever()

Answer 1

It seems the behavior of cancel and shield is not what I would expect. If I have a task wrapped in shield and I cancel it, the await-ing coroutine returns from that await statement immediately rather than awaiting for the task to finish as shield would suggest. Additionally, the task that was run with shield continues to run but its future is now cancelled an not await-able.

Conceptually shield is like a bullet-proof vest that absorbs the bullet and protects the wearer, but is itself destroyed by the impact. shield absorbs the cancellation, and reports itself as canceled, raising a CancelledError when asked for result, but allows the protected task to continue running. (Artemiy's answer explains the implementation.)

Cancellation of the future returned by shield could have been implemented differently, e.g. by completely ignoring the cancel request. The current approach ensures that the cancellation "succeeds", i.e. that the canceller can't tell that the cancellation was in fact circumvented. This is by design, and it makes the cancellation mechanism more consistent on the whole.

What is the proper method to shield a task from cancellation and then wait for it to complete before returning

By keeping two objects: the original task, and the shielded task. You pass the shielded task to whatever function it is that might end up canceling it, and you await the original one. For example:

async def coro():
    print('starting')
    await asyncio.sleep(2)
    print('done sleep')

async def cancel_it(some_task):
    await asyncio.sleep(0.5)
    some_task.cancel()
    print('cancellation effected')

async def main():
    loop = asyncio.get_event_loop()
    real_task = loop.create_task(coro())
    shield = asyncio.shield(real_task)
    # cancel the shield in the background while we're waiting
    loop.create_task(cancel_it(shield))
    await real_task

    assert not real_task.cancelled()
    assert shield.cancelled()

asyncio.get_event_loop().run_until_complete(main())

The code waits for the task to fully complete, despite its shield getting cancelled.