Can I change the connection pool size for Python's "requests" module?

Question

(edit: Perhaps I am wrong in what this error means. Is this indicating that the connection pool at my CLIENT is full? or a connection pool at the SERVER is full and this is the error my client is being given?)

I am attempting to make a large number of http requests concurrently using the python threading and requests module. I am seeing this error in logs:

WARNING:requests.packages.urllib3.connectionpool:HttpConnectionPool is full, discarding connection: 

What can I do to increase the size of the connection pool for requests?

Answer 1

This should do the trick:

import requests.adapters  session = requests.Session() adapter = requests.adapters.HTTPAdapter(pool_connections=100, pool_maxsize=100) session.mount('http://', adapter) response = session.get("/mypage") 

Answer 2

Note: Use this solution only if you cannot control the construction of the connection pool (as described in @Jahaja's answer).

The problem is that the urllib3 creates the pools on demand. It calls the constructor of the urllib3.connectionpool.HTTPConnectionPool class without parameters. The classes are registered in urllib3 .poolmanager.pool_classes_by_scheme. The trick is to replace the classes with your classes that have different default parameters:

def patch_http_connection_pool(**constructor_kwargs):     """     This allows to override the default parameters of the      HTTPConnectionPool constructor.     For example, to increase the poolsize to fix problems      with "HttpConnectionPool is full, discarding connection"     call this function with maxsize=16 (or whatever size      you want to give to the connection pool)     """     from urllib3 import connectionpool, poolmanager      class MyHTTPConnectionPool(connectionpool.HTTPConnectionPool):         def __init__(self, *args,**kwargs):             kwargs.update(constructor_kwargs)             super(MyHTTPConnectionPool, self).__init__(*args,**kwargs)     poolmanager.pool_classes_by_scheme['http'] = MyHTTPConnectionPool 

Then you can call to set new default parameters. Make sure this is called before any connection is made.

patch_http_connection_pool(maxsize=16) 

If you use https connections you can create a similar function:

def patch_https_connection_pool(**constructor_kwargs):     """     This allows to override the default parameters of the     HTTPConnectionPool constructor.     For example, to increase the poolsize to fix problems     with "HttpSConnectionPool is full, discarding connection"     call this function with maxsize=16 (or whatever size     you want to give to the connection pool)     """     from urllib3 import connectionpool, poolmanager      class MyHTTPSConnectionPool(connectionpool.HTTPSConnectionPool):         def __init__(self, *args,**kwargs):             kwargs.update(constructor_kwargs)             super(MyHTTPSConnectionPool, self).__init__(*args,**kwargs)     poolmanager.pool_classes_by_scheme['https'] = MyHTTPSConnectionPool