Using requests module, how to handle 'set-cookie' in request response?

Question

I'm attempting to open a login page (GET), fetch the cookies provided by the webserver, then submit a username and password pair to log into the site (POST).

Looking at this Stackoverflow question/answer, I would think that I would just do the following:

import requests import cookielib   URL1 = 'login prompt page' URL2 = 'login submission URL' jar = cookielib.CookieJar()  r = requests.get(URL1, cookies=jar) r2 = requests.post(URL2, cookies=jar, data="username and password data payload") 

However, in r there is a set-cookie in the header, but that isn't changing in the jar object. In fact, nothing is being populated into jar as the linked question's response would indicate.

I'm getting around this in my code by having a headers dict and after doing the GET or POST, using this to handle the set-cookie header:

headers['Cookie'] = r.headers['set-cookie'] 

Then passing around the header in the requests methods. Is this correct, or is there a better way to apply the set-cookie?

Answer 1

Ignore the cookie-jar, let requests handle cookies for you. Use a session object instead, it'll persist cookies and send them back to the server:

with requests.Session() as s:     r = s.get(URL1)     r = s.post(URL2, data="username and password data payload") 

Answer 2

There's an included class called a session which automatically handles this sort of thing for you. You can create an instance of it, and then call get and set right on that instance instead.

import requests  URL1 = 'login prompt page' URL2 = 'login submission URL'  session = requests.Session()  r = session.get(URL1) r2 = session.post(URL2, data="username and password data payload")