Throttling a http client in Twisted

Question

Context

(In case I am asking the wrong question)

I have clients and a server. The applications they use is written in Twisted. The server needs a simple API that the clients can use. It must

• Allow an approximation of function calls where at most one argument or the response is a stream of binary data
• Support authentication

However the client must also be able to throttle their read/write rate and number of connections to the server.

I reached for the Twisted HTTP libraries as although HTTP is more complex than I need, I assumed it would do the job as I assumed it could be combined WrappingFactories such as twisted.protocols.policies.ThrottlingFactory and twisted.protocols.policies.LimitConnectionsByPeer

Problem

There seems to be no easy way to wrap the _HTTP11ClientFactory used by the Agent. For good reason as when I tried adding a hacky method of wrapping the factory it resulted in errors.

Question(s)

• Am I just missing something - is there an easy way to throttle http requests on the client side?
• If not is there another protocol I can use that would support my usecase, or is it better to write something simple that I can easily understand and apply the aforementioned factories to?

Answer 1

Throttling the rate of HTTP requests involves more than what ThrottlingFactory or LimitConnectionsByPeer do.

LimitConnectionsByPeer is for limiting the rate of incoming connections, to a server, and therefore not very helpful for clients.

ThrottlingFactory ought to work, although you might have to do some extra work to hack it in there, and that is probably not documented very well. It would be useful for you to include the hacky code you used and the error you got if you would like more help with that. However, what it does is limit the line rate of that connection, not the rate of outgoing connections. For that, you would need a client endpoint that would delay outgoing connection success.

As far as I know though, you are not missing anything. This probably isn't quite as hard as it seems, but there is nothing out of the box that would just do it for you. And you're right that this is something that Twisted's HTTP client should do.