Python 3.6 async GET requests in with aiohttp are running synchronously

Question

I have the below functioning properly, but for some reason the requests seem to be executing synchronously, instead of asynchronously.

My assumption now is that this is happening because of the for record in records for loop in the main function, but i'm not sure how to change this so that requests can execute async. If this is not the case, what else would I need to change?

async def do_request(query_string):
        base_url = 'https://maps.googleapis.com/maps/api/place/textsearch/json?'
        params = {'key': google_api_key,
                  'query': query_string}
        async with aiohttp.ClientSession() as session:
            async with session.request('GET', base_url, params=params) as resp:
                return resp


async def main():
    create_database_and_tables()
    records = prep_sample_data()[:100]

    for record in records:
        r = Record(record)

        if not r.is_valid:
            continue

        query_string = r.generate_query_string()

        resp = await do_request(query_string)
        print("NOW WRITE TO DATABASE")

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Answer 1

You are awaiting on separate do_request() calls in sequence. Instead of awaiting on them directly (which blocks until the coroutine is done), use the asyncio.gather() function to have the event loop run them concurrently:

async def main():
    create_database_and_tables()
    records = prep_sample_data()[:100]

    requests = []
    for record in records:
        r = Record(record)

        if not r.is_valid:
            continue

        query_string = r.generate_query_string()

        requests.append(do_request(query_string))

    for resp in asyncio.gather(*requests):
        print("NOW WRITE TO DATABASE")

The asyncio.gather() return value is a list of all the results the coroutines returned, in the same order you passed them to the gather() function.

If you needed the original records to process the responses, you can pair up record and query string in several different ways:

• store valid records in a separate list and use zip() to pair them up again as you process the responses
• use a helper coroutine that takes the valid record, produces a query string, invokes the request, and returns the record and response together as a tuple.

You can also mix in the response handling into a gathered coroutine; one that takes a record, produces the query string, awaits on do_request and then stores the result in the database when the response is ready.

In other words, divide up your work that needs to happen consecutively, in coroutines and gather those.