Why do I get the error "ClientConnectorError"?

Question

I have an asynchronous function to get data from the site:

async def get_matches_info(url):
    async with aiohttp.ClientSession() as session:
        try:
            async with session.get(url, proxy=proxy) as response:
                ...
                ...
                ...
                ...
        except:
            print('ERROR GET URL: ', url)
            print(traceback.print_exc())

I have a list of about 200 links. Almost always everything is OK, but sometimes I get the following error:

Traceback (most recent call last):
  File "C:\Python37\lib\site-packages\aiohttp\connector.py", line 924, in _wrap_create_connection
    await self._loop.create_connection(*args, **kwargs))
  File "C:\Python37\lib\asyncio\base_events.py", line 986, in create_connection
    ssl_handshake_timeout=ssl_handshake_timeout)
  File "C:\Python37\lib\asyncio\base_events.py", line 1014, in _create_connection_transport
    await waiter
ConnectionResetError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "parser.py", line 90, in get_matches_info
    async with session.get(url, proxy=proxy) as response:
  File "C:\Python37\lib\site-packages\aiohttp\client.py", line 1005, in __aenter__
    self._resp = await self._coro
  File "C:\Python37\lib\site-packages\aiohttp\client.py", line 476, in _request
    timeout=real_timeout
  File "C:\Python37\lib\site-packages\aiohttp\connector.py", line 522, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "C:\Python37\lib\site-packages\aiohttp\connector.py", line 851, in _create_connection
    req, traces, timeout)
  File "C:\Python37\lib\site-packages\aiohttp\connector.py", line 1085, in _create_proxy_connection
    req=req)
  File "C:\Python37\lib\site-packages\aiohttp\connector.py", line 931, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host www.myscore.com.ua:443 ssl:None [None]

I checked all the links from the errors - they are working. Why can this happen?

Answer 1

This is probably the server limit of concurrent requests thinking you are doing a DoS attack. If you are in control of the server and it's running Apache you can raise that limit on the httpd conf for MaxKeepAliveRequests. If not, you can also set a limit to the amount of concurrent asyncio requests by using its semaphores. The example below sets that limit to 100 concurrent requests.

async def get_matches_info(url):
    sem = asyncio.Semaphore(100)
    async with sem:
        async with aiohttp.ClientSession() as session:
            try:
                async with session.get(url, proxy=proxy) as response:
                    ...

Note that if you call this function recursively that semaphore queue will be reset each time so you might want to consider placing this semaphore outside the function and pass it as a parameter.