Python urllib3: close idle connection after some time

Question

Is there a way to tell Python urllib3 to not reuse idle connections after some period of time, and instead to close them?

Looking in https://urllib3.readthedocs.io/en/latest/reference/index.html#module-urllib3.connectionpool doesn't seem to show anything relevant.

Answer 1

Remember:

A connection pool is a cache of database connections maintained so that the connections can be "reused" when future requests to the database are required.

You can do this is many ways (I guess):

• Set retries to one.

This breaks your connection if it failed one time. To set it:

import requests
s = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=1) # is zero for default
s.mount('http://', a)

---

• Change pool connections.

The "pool_connections" is the number of host-pools to keep around. For example, if you're connecting to 100 different hosts, and pool_connections=10, then only the latest 10 hosts' connections will be re-used. To set that:

s = requests.Session()
s.mount('https://', HTTPAdapter(pool_connections=1))
s.get('https://www.example.com')

This will stop the reuse of pools.

---

• Pool Maxsize

This is cared only if you use Session in a multithreaded environment. To set it:

s = requests.Session()
s.mount('https://', HTTPAdapter(pool_connections=1, pool_maxsize=1))

---

• Configure Maxsize

he :class:~connectionpool.ConnectionPool class keeps a pool of individual :class:~connection.HTTPConnection instances. These connections are used during an individual request and returned to the pool when the request is complete. By default only one connection will be saved for re-use. To set it (it is, by default):

from urllib3 import HTTPConnectionPool
pool = HTTPConnectionPool('www.example.com', maxsize=0) #likely to slow you down cuz it never stores the pools

maxsize – Number of connections to save that can be reused. More than 1 is useful in multithreaded situations.

---

• Let the Pool Manager do that!

The PoolManager uses a Least Recently Used (LRU) policy for discarding old pools. That is, if you set the PoolManager num_pools to 10, then after making requests to 11 or more different hosts, the least recently used pools will be cleaned up eventually. So to do that:

from urllib3 import PoolManager
manager = PoolManager(1) # not the manager cleans up pools used for one time
r = manager.request('GET', 'http://www.example.com/')

Also, the docs says that:

Cleanup of stale pools does not happen immediately.

So for that Use RecentlyUsedContainer (Docs contains only one line).

Note:

Setting arguments if PoolManager affects all the pools connected thereby.

---

Hopes this helps you. Get the advanced usage docs HERE .