Throttling bandwidth of an SSL connection

Question

I asked a question about how to throttle a python upload, which sent me to this answer, where I was informed of a little helper library called socket-throttle. That's all fine and dandy for regular HTTP and probably also for most plain uses of the socket. However, I'm trying to throttle an SSL connection, and trying to combine socket-throttle with the stock SSL library (used implicitly by requests) causes an exception deep in the guts of the library:

  File "***.py", line 590, in request
    r = self.session.get(url, headers=extra_headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 394, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 382, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 485, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 324, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 478, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 285, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py", line 95, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util.py", line 643, in ssl_wrap_socket
    ssl_version=ssl_version)
  File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 211, in __init__
    socket.__init__(self, _sock=sock._sock)
  File "***/socket_throttle.py", line 54, in __getattr__
    return getattr(self._wrappedsock, attr)
AttributeError: '_socket.socket' object has no attribute '_sock'

Well, that's a downer. As you can tell, the ssl package is trying to use one of the socket's private fields, _sock rather than the socket itself. (Isn't the point of private fields that you're not supposed to access them from the outside? Grr.) If I try to inject myself into that field on my ThrottledSocket object, I run into this problem:

    File "/home/alex/dev/jottalib/src/jottalib/JFS.py", line 590, in request
    r = self.session.get(url, headers=extra_headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 394, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 382, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 485, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 324, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 478, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 285, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.py", line 95, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util.py", line 643, in ssl_wrap_socket
    ssl_version=ssl_version)
  File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python2.7/ssl.py", line 241, in __init__
    ciphers)
TypeError: must be _socket.socket, not ThrottledSocket

Now what? Is there somewhere else in this where I could rate-limit the python communication? Or is there a cleaner way to do it than having to override the socket implementation? Which turns out to be moot anyway, since the ssl package just tries to bypass it altogether.

Answer 1

Depending on your requirements, you can and maybe should solve this particular problem on the OS level instead of on the application level.

Approaching this on the OS level has two advantages. First, it does not make a difference how the sockets involved are used (HTTP or HTTPS or IRC or some ping of death packets -- it does not matter). Secondly, the more you decouple the different components of your system, the easier it is to make changes afterwards and to debug issues.

There are tools (at least for POSIX-compliant systems) for throttling bandwidth of network interfaces and/or processes. You might want to have a look at these, for example:

• trickle (for shaping traffic of processes)
• wondershaper (for shaping traffic of entire network interfaces, I have actually used this from within a modern Ubuntu, and it works perfectly fine)

These discussions might be relevant for you:

• https://superuser.com/questions/66574/how-to-throttle-bandwidth-on-a-linux-network-interface
• http://jwalanta.blogspot.de/2009/04/easy-bandwidth-shaping-in-linux.html
• https://unix.stackexchange.com/questions/28198/how-to-limit-network-bandwidth

Answer 2

It looks like you're trying to throttle HTTP requests. If that's the case, you can try RequestsThrottler instead. Python requests is way nicer than httplib too.