Maximum number of connections per host with twisted.web.client.Agent

Question

I have the following code which creates an HTTPConnectionPool using TwistedMatrix Python framework, and an Agent for HTTP requests:

    self.pool = HTTPConnectionPool(reactor, persistent=True)
    self.pool.retryAutomatically = False
    self.pool.maxPersistentPerHost = 1
    self.agent = Agent(reactor, pool=self.pool)

then I create requests to connect to a local server:

    d = self.agent.request(
        "GET",
         url,
         Headers({"Host": ["localhost:8333"]}),
         None)

The problem is: the local server sometimes behaves incorrectly when multiple simultaneous requests are made, so I would like to limit the number of simultaneous requests to 1.

The additional requests should be queued until the pending request completes.

I've tried with self.pool.maxPersistentPerHost = 1 but it doesn't work.

Does twisted.web.client.Agent with HTTPConnectionPool support limiting the maximum number of connections per host, or do I have to implement a request FIFO queue myself?

Answer 1

The reason setting maxPersistentPerHost to 1 didn't help is that maxPersistentPerHost is for controlling the maximum number of persistent connections to cache per host. It does not prevent additional connections from being opened in order to service new requests, it will only cause them to be closed immediately after a response is received, if the maximum number of cached connections has already been reached.

You can enforce serialization in a number of ways. One way to have a "FIFO queue" is with twisted.internet.defer.DeferredLock. Use it together with Agent like this:

lock = DeferredLock()
d1 = lock.run(agent.request, url, ...)
d2 = lock.run(agent.request, url, ...)

The second request will not run until after the first as completed.