How can I wait for an object's __del__ to finish before the async loop closes?

Question

I have a class that will have an aiohttp.ClientSession object in it.

Normally when you use

async with aiohttp.ClientSession() as session:  
   # some code

The session will close since the session's __aexit__ method is called.

I cant use a context manager since I want to keep the session persistent for the entire lifetime of the object.

This works:

import asyncio
import aiohttp

class MyAPI:
    def __init__(self):
        self.session = aiohttp.ClientSession()

    def __del__(self):
        # Close connection when this object is destroyed
        print('In __del__ now')
        asyncio.shield(self.session.__aexit__(None, None, None))



async def main():
    api = MyAPI()

asyncio.run(main())

However if in some place an exception is raised, the event loop is closed before the __aexit__ method is finished. How can I overcome this?

stacktrace:

Traceback (most recent call last):
  File "/home/ron/.PyCharm2018.3/config/scratches/async.py", line 19, in <module>
    asyncio.run(main())
  File "/usr/local/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 568, in run_until_complete
    return future.result()
  File "/home/ron/.PyCharm2018.3/config/scratches/async.py", line 17, in main
    raise ValueError
ValueError
In __del__ now
Exception ignored in: <function MyAPI.__del__ at 0x7f49982c0e18>
Traceback (most recent call last):
  File "/home/ron/.PyCharm2018.3/config/scratches/async.py", line 11, in __del__
  File "/usr/local/lib/python3.7/asyncio/tasks.py", line 765, in shield
  File "/usr/local/lib/python3.7/asyncio/tasks.py", line 576, in ensure_future
  File "/usr/local/lib/python3.7/asyncio/events.py", line 644, in get_event_loop
RuntimeError: There is no current event loop in thread 'MainThread'.
sys:1: RuntimeWarning: coroutine 'ClientSession.__aexit__' was never awaited
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f49982c2e10>

Answer 1

Don't use a __del__ hook to clean up asynchronous resources. You can't count it being called at all, let alone control when it'll be used or if the async loop is still available at that time. You really want to handle this explicitly.

Either make the API an async context manager, or otherwise explicitly clean up resources at exit, with a finally handler, say; the with and async with statements are basically designed to encapsulate resource cleanup traditionally handled in finally blocks.

I'd make the API instance a context manager here:

class MyAPI:
    def __init__(self):
        self.session = aiohttp.ClientSession()

    async def __aenter__(self):
        return self

    async def __aexit__(self, *excinfo):
        await self.session.close()

Note that all that ClientSession.__aexit__() really does is await on self.close(), so the above goes straight to that coroutine.

Then use this in your main loop with:

async def main():
    async with MyAPI() as api:
        pass

Another option is to supply your own session object to the MyAPI instance and take responsibility yourself for closing it when you are done:

class MyAPI:
    def __init__(self, session):
        self.session = session

async def main():
    session = aiohttp.ClientSession()
    try:
        api = MyAPI(session)
        # do things with the API
    finally:
        await session.close()

Answer 2

As @Martijn Pieters said, you can't force the event loop to wait for an object's __del__ destructor call. However, you can still use the __del__ destructor to close asynchronous resources by first checking if the loop is running and starting a new loop if it's not. For example, the asyncio Redis module uses this technique when destructing its Client class. For your code, specifically, the the destructor would be as follows:

import asyncio
import aiohttp


class MyAPI:

    def __init__(self):
        self.session = aiohttp.ClientSession()

    def __del__(self):
        # Close connection when this object is destroyed
        try:
            loop = asyncio.get_event_loop()
            if loop.is_running():
                loop.create_task(self.session.close())
            else:
                loop.run_until_complete(self.session.close())
        except Exception:
            pass