requests, cannot assign requested address, out of ports?

Question

requests.exceptions.ConnectionError: ('Connection aborted.', error(99, 'Cannot assign requested address'))

I was getting this error when running multiple processes utilizing the python requests library and calling the post function to an API that was returning very quickly (<10ms).

Dialing down the number of processes running had a delaying effect, but only dialing down to 1 process eliminated the problem. This was not a solution, but did indicate a finite resource was the culprit.

Answer 1

The way I resolved my issue was to use the requests.Session class where I would reuse the same connection/session for each call in a given process.

Contrived example:

import requests
for line in file:
  requests.get('http://example.com/api?key={key}'.format(key=line['key']))

becomes

import requests
with requests.Session() as session:
  for line in file:
    session.get('http://example.com/api?key={key}'.format(key=line['key']))

These questions had some related advice:

Repeated POST request is causing error "socket.error: (99, 'Cannot assign requested address')" Python urllib2: Cannot assign requested address

Answer 2

I too faced similar issue while executing multiple POST statements using python's request library in Spark. To make it worse, I used multiprocessing over each executor to post to a server. So thousands of connections created in seconds that took few seconds each to change the state from TIME_WAIT and release the ports for the next set of connections.

Out of all the available solutions over the internet that speak of disabling keep-alive, using with request.Session() et al., I found this answer to be working. You may need to put the header content in a separate line outside the post command though.

headers = {
        'Connection': 'close'
}
with requests.Session() as session:
response = session.post('https://xx.xxx.xxx.x/xxxxxx/x', headers=headers, files=files, verify=False)
results = response.json()
print results