how to use requests.session so that headers are presevred and reused in subsequent get requests

Question

I might have misunderstood the requests.session object.

headers ={'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.28 Safari/537.36'}
s = requests.Session()
r = s.get('https://www.barchart.com/', headers = headers)
print(r.status_code)

This works fine and return 200 as expected.

However this following return 403 and shows that the headers from the first request have not been saved in the session like it would manually using a browser:

headers ={'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.28 Safari/537.36'}
s = requests.Session()
r = s.get('https://www.barchart.com/', headers = headers)
r = s.get('https://www.barchart.com/futures/quotes/CLQ20')
print(r.status_code)
print(s.headers)

I thought there would be a way to compound headers, cookies etc from 1 requests to the other using the session object... am i wrong ?

Answer 1

You can use session.headers (doc) property to specify headers that are sent with each request:

import requests

headers ={'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.28 Safari/537.36'}

s = requests.session()
s.headers = headers       # <-- set default headers here

r = s.get('https://www.barchart.com/')
print(r.status_code)
print(s.headers)
print('-' * 80)
r = s.get('https://www.barchart.com/futures/quotes/CLQ20')
print(r.status_code)
print(s.headers)

s.close()

Prints:

200
{'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.28 Safari/537.36'}
--------------------------------------------------------------------------------
200
{'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.28 Safari/537.36'}