ConnectionError, Max retries exceeded with url, (Caused by None)

Question

What does it mean when it says:

requests.exceptions.ConnectionError: None: Max retries exceeded with url: /myurl (Caused by None)

Specifically what does "Caused by None" mean?

I have a python client and a simple clojure server running on the same machine. The server runs on compojure+http-kit with 4 threads. The client continuously submit POST requests using 3 to 4 processes using multiprocessing.Pool with 3 worker processes.

Every so often, the client would die with the ConnectionError described above. I have settings retries=3 on the client side and increasing the queue size on the server to 1000000 with no effect.

Any help would be appreciated.

edit: A correction, I am actually sending POST requests not GET.

The actual script is too large to post here but basically it works like this. I have a function that calls post with some data:

def post_func(my_data, config):
    return requests.post(my_url, data=json.dumps({"data": my_data, "config": config}))

A class that wraps multiprocessing.Pool:

class Executor(object):
    def __init__(self, nprocs):
        self.pool = Pool(processes=nprocs)

    def execute(self, func, list_of_data):
        return self.pool.map(func, list_of_data)

Another function that calls Executor.execute() with different configurations:

function eval(executor, list_of_data, config):
    start = timer()
    func = partial(post_func, config=config)
    results = executor.execute(func, list_of_data)
    taken = timer()-start
    return results

A single Executor is reused for all eval() calls. eval() is then wrapped in a scoring function and given to pyswarm to optimise:

pso(score_func, lbs, ubs, swarmsize=20, maxiter=20, debug=True)

edit: something I probably should have done earlier, but catching the ConnectionError properly gives me this:

ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /my_url (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f0ccb4b5550>: Failed to establish a new connection: [Errno 99] Cannot assign requested address',))

I have now rewrote the script to reuse a single requests.Session, will know soon if this fixes the problem.

Answer 1

Reusing a single request session fixed the problem for me.