Collecting results from python coroutines before loop finishes

Question

I have a python discord bot built with discord.py, meaning the entire program runs inside an event loop.

The function I'm working on involves making several hundred HTTP requests and add the results to a final list. It takes about two minutes to do these in order, so I'm using aiohttp to make them async. The related parts of my code are identical to the quickstart example in the aiohttp docs, but it's throwing a RuntimeError: Session is closed. The methodology was taken from an example at https://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html under 'Fetch multiple URLs'.

async def searchPostList(postUrls, searchString)
    futures = []
    async with aiohttp.ClientSession() as session:
        for url in postUrls:
            task = asyncio.ensure_future(searchPost(url,searchString,session))
            futures.append(task)

    return await asyncio.gather(*futures)


async def searchPost(url,searchString,session)):
    async with session.get(url) as response:
        page = await response.text()

    #Assorted verification and parsing
    Return data

I don't know why this error turns up since my code is so similar to two presumably functional examples. The event loop itself is working fine. It runs forever, since this is a bot application.

Answer 1

In the example you linked, the gathering of results was within the async with block. If you do it outside, there's no guarantee that the session won't close before the requests are even made!

Moving your return statement inside the block should work:

async with aiohttp.ClientSession() as session:
        for url in postUrls:
            task = asyncio.ensure_future(searchPost(url,searchString,session))
            futures.append(task)

        return await asyncio.gather(*futures)