python urllib2 timing

Question

I'd like to collect statistics related to how long each phase of a web request takes. httplib offers:

def run(self):
    conn = httplib.HTTPConnection('www.example.com')
    start = time.time()
    conn.request('GET', '/')
    request_time = time.time()
    resp = conn.getresponse()
    response_time = time.time()
    conn.close()
    transfer_time = time.time()

    self.custom_timers['request sent'] = request_time - start
    self.custom_timers['response received'] = response_time - start
    self.custom_timers['content transferred'] = transfer_time - start

    assert (resp.status == 200), 'Bad Response: HTTP %s' % resp.status

Are these statistics available from a more high-level interface like urllib2? Is there high level library offering such statistics?

Answer 1

As mentioned in a related question a good way to do this now is to use the requests library. You can use it to measure the request latency, though I'm not sure if you can measure the content transfer timing. You could potentially do that by comparing a HEAD request to a GET request.