Bandwidth throttling in Python

Question

What libraries out there let you control the download speed of network requests (http in particular). I don't see anything built-in in urllib2 (nor in (Py)Qt which I intend on using).

Can Twisted control bandwidth? If not, how can I control the read buffer size of urllib2 or Twisted? sleeping to suspend network operations isn't an option.

Answer 1

urllib2 doesn't offer a way to do this, so you'd have to extend some of the classes it uses and implement rate limiting yourself. You might want to look at this question. If you decide to write a limiter, read up on the token bucket and leaky bucket algorithms.

Some attempted hack solutions available on github are Phredward/throttle and minkustree/socket-throttle.

Alternatively, you could use pycurl along with the CURLOPTMAXRECVSPEEDLARGE option.

EDIT: The urlgrabber package appears to support throttling as well, and is probably easier to understand than pycurl.

If you prefer to program using an event loop model, there's the Twisted approach, which has already been mentioned in another answer.

Answer 2

Of course twisted can. You want twisted.protocols.policies.ThrottlingFactory. Just wrap your existing factory in it before you pass it to whatever wants a factory.