How to Speed Up Python's urllib2 when doing multiple requests

Question

I am making several http requests to a particular host using python's urllib2 library. Each time a request is made a new tcp and http connection is created which takes a noticeable amount of time. Is there any way to keep the tcp/http connection alive using urllib2?

Answer 1

If you switch to httplib, you will have finer control over the underlying connection.

For example:

import httplib  conn = httplib.HTTPConnection(url)  conn.request('GET', '/foo') r1 = conn.getresponse() r1.read()  conn.request('GET', '/bar') r2 = conn.getresponse() r2.read()  conn.close() 

This would send 2 HTTP GETs on the same underlying TCP connection.