Keeping a session in python while making HTTP requests

Question

I need to write a python script that makes multiple HTTP requests to the same site. Unless I'm wrong (and I may very well be) urllib reauthenticates for every request. For reasons I won't go into I need to be able to authenticate once and then use that session for the rest of my requests.

I'm using python 2.3.4

Answer 1

Use Requests library. From http://docs.python-requests.org/en/latest/user/advanced/#session-objects :

The Session object allows you to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance.

s = requests.session()  s.get('http://httpbin.org/cookies/set/sessioncookie/123456789') r = s.get("http://httpbin.org/cookies")  print r.text # '{"cookies": {"sessioncookie": "123456789"}}' 

Answer 2

If you want to keep the authentication you need to reuse the cookie. I'm not sure if urllib2 is available in python 2.3.4 but here is an example on how to do it:

req1 = urllib2.Request(url1) response = urllib2.urlopen(req1) cookie = response.headers.get('Set-Cookie')  # Use the cookie is subsequent requests req2 = urllib2.Request(url2) req2.add_header('cookie', cookie) response = urllib2.urlopen(req2)