Repeated POST request is causing error "socket.error: (99, 'Cannot assign requested address')"

Question

I have a web-service deployed in my box. I want to check the result of this service with various input. Here is the code I am using:

import sys
import httplib
import urllib

apUrl = "someUrl:somePort"

fileName = sys.argv[1]
conn = httplib.HTTPConnection(apUrl)

titlesFile = open(fileName, 'r')

try:
    for title in titlesFile:

        title = title.strip()
        params = urllib.urlencode({'search': 'abcd', 'text': title})
        conn.request("POST", "/somePath/", params)
        response = conn.getresponse()
        data = response.read().strip()
        print data+"\t"+title

        conn.close()

finally:
    titlesFile.close()

This code is giving an error after same number of lines printed (28233). Error message:

Traceback (most recent call last):
  File "testService.py", line 19, in ?
    conn.request("POST", "/somePath/", params)
  File "/usr/lib/python2.4/httplib.py", line 810, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.4/httplib.py", line 833, in _send_request
    self.endheaders()
  File "/usr/lib/python2.4/httplib.py", line 804, in endheaders
    self._send_output()
  File "/usr/lib/python2.4/httplib.py", line 685, in _send_output
    self.send(msg)
  File "/usr/lib/python2.4/httplib.py", line 652, in send
    self.connect()
  File "/usr/lib/python2.4/httplib.py", line 636, in connect
    raise socket.error, msg
socket.error: (99, 'Cannot assign requested address')

I am using Python 2.4.3. I am doing conn.close() also. But why is this error being given?

Answer 1

This is not a python problem.

In linux kernel 2.4 the ephemeral port range is from 32768 through 61000. So number of available ports = 61000-32768+1 = 28233. From what i understood, because the web-service in question is quite fast (<5ms actually) thus all the ports get used up. The program has to wait for about a minute or two for the ports to close.

What I did was to count the number of conn.close(). When the number was 28000 wait for 90sec and reset the counter.

Answer 2

BIGYaN identified the problem correctly and you can verify that by calling "netstat -tn" right after the exception occurs. You will see very many connections with state "TIME_WAIT".

The alternative to waiting for port numbers to become available again is to simply use one connection for all requests. You are not required to call conn.close() after each call of conn.request(). You can simply leave the connection open until you are done with your requests.

Answer 3

I too faced similar issue while executing multiple POST statements using python's request library in Spark. To make it worse, I used multiprocessing over each executor to post to a server. So thousands of connections created in seconds that took few seconds each to change the state from TIME_WAIT and release the ports for the next set of connections.

Out of all the available solutions available over the internet that speak of disabling keep-alive, using with request.Session() et al, I found this answer to be working which makes use of 'Connection' : 'close' configuration as header parameter. You may need to put the header content in a separte line outside the post command though.

headers = {
        'Connection': 'close'
}
with requests.Session() as session:
response = session.post('https://xx.xxx.xxx.x/xxxxxx/x', headers=headers, files=files, verify=False)
results = response.json()
print results

This is my answer to the similar issue using the above solution.