Error when loading cookies into a Python request session

Question

I am trying to load cookies into my request session in Python from selenium exported cookies, however when I do it returns the following error: "'list' object has no attribute 'extract_cookies'"

def load_cookies(filename):
    with open(filename, 'rb') as f:
        return pickle.load(f)

initial_state= requests.Session()
initial_state.cookies=load_cookies(time_cookie_file)
search_requests = initial_state.get(search_url)

Everywhere I see this should work, however my cookies are a list of dictionaries, which is what I understand all cookies are, and why I assume this works with Selenium. However for some reason it does not work with requests, any and all help in this regard would be really great, it feels like I am missing something obvious!

Cookies have been dumped from Selenium using:

with open("Filepath.pkl", 'wb') as f:
    pickle.dump(driver.get_cookies(), f)

An example of the cookies would be (slightly obfuscated):

[{'domain': '.website.com',
  'expiry': 1640787949,
  'httpOnly': False,
  'name': '_ga',
  'path': '/',
  'secure': False,
  'value': 'GA1.2.1111111111.1111111111'},
 {'domain': 'website.com',
  'expiry': 1585488346,
  'httpOnly': False,
  'name': '__pnahc',
  'path': '/',
  'secure': False,
  'value': '0'}]

I have now managed to load in the cookies as per the answer below, however it does not seem like the cookies are loaded in properly as they do not remember anything, however if I load the cookies in when browsing through Selenium they work fine.

Answer 1

Cookie

The Cookie HTTP request header contains stored HTTP cookie previously sent by the server with the Set-Cookie header. A HTTP cookie is a small piece of data that a server sends to the user's web browser. The browser may store the cookies and send it back with the next request to the same server. Typically, cookies to tell if two requests came from the same browser, keeping the user logged in.

---

Demonstration using Selenium

To demonstrate the usage of cookies using Selenium we have stored the cookies using pickle once the user had logged into the website http://demo.guru99.com/test/cookie/selenium_aut.php. In the next step, we opened the same website, adding the cookies and was able to land as a logged in user.

• Code Block to store the cookies:

  from selenium import webdriver
  import pickle
  
  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get('http://demo.guru99.com/test/cookie/selenium_aut.php')
  driver.find_element_by_name("username").send_keys("abc123")
  driver.find_element_by_name("password").send_keys("123xyz")
  driver.find_element_by_name("submit").click()
  pickle.dump( driver.get_cookies() , open("cookies.pkl","wb"))
  

• Code Block to use the stored cookies for automatic authentication:

  from selenium import webdriver
  import pickle
  
  options = webdriver.ChromeOptions() 
  options.add_argument("start-maximized")
  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
  driver.get('http://demo.guru99.com/test/cookie/selenium_aut.php')
  cookies = pickle.load(open("cookies.pkl", "rb"))
  for cookie in cookies:
      driver.add_cookie(cookie)
  driver.get('http://demo.guru99.com/test/cookie/selenium_cookie.php')
  

---

Demonstration using Requests

To demonstrate usage of cookies using session and requests we have accessed the site https://www.google.com, added a new dictionary of cookies:

{'name':'my_own_cookie','value': 'debanjan' ,'domain':'.stackoverflow.com'}

Next, we have used the same requests session to send another request which was successful as follows:

• Code Block:

  import requests
  
  s1 = requests.session()
  s1.get('https://www.google.com')
  print("Original Cookies")
  print(s1.cookies)
  print("==========")
  cookie = {'name':'my_own_cookie','value': 'debanjan' ,'domain':'.stackoverflow.com'}
  s1.cookies.update(cookie)
  print("After new Cookie added")
  print(s1.cookies)
  

• Console Output:

  Original Cookies
  <RequestsCookieJar[<Cookie 1P_JAR=2020-01-21-14 for .google.com/>, <Cookie NID=196=NvZMMRzKeV6VI1xEqjgbzJ4r_3WCeWWjitKhllxwXUwQcXZHIMRNz_BPo6ujQduYCJMOJgChTQmXSs6yKX7lxcfusbrBMVBN_qLxLIEah5iSBlkdBxotbwfaFHMd-z5E540x02-YZtCm-rAIx-MRCJeFGK2E_EKdZaxTw-StRYg for .google.com/>]>
  ==========
  After new Cookie added
  <RequestsCookieJar[<Cookie domain=.stackoverflow.com for />, <Cookie name=my_own_cookie for />, <Cookie value=debanjan for />, <Cookie 1P_JAR=2020-01-21-14 for .google.com/>, <Cookie NID=196=NvZMMRzKeV6VI1xEqjgbzJ4r_3WCeWWjitKhllxwXUwQcXZHIMRNz_BPo6ujQduYCJMOJgChTQmXSs6yKX7lxcfusbrBMVBN_qLxLIEah5iSBlkdBxotbwfaFHMd-z5E540x02-YZtCm-rAIx-MRCJeFGK2E_EKdZaxTw-StRYg for .google.com/>]>
  

---

Conclusion

Clearly, the newly added dictionary of cookies {'name':'my_own_cookie','value': 'debanjan' ,'domain':'.stackoverflow.com'} is pretty much in use within the second request.

---

Passing Selenium Cookies to Python Requests

Now, if your usecase is to passing Selenium Cookies to Python Requests, you can use the following solution:

from selenium import webdriver
import pickle
import requests

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('http://demo.guru99.com/test/cookie/selenium_aut.php')
driver.find_element_by_name("username").send_keys("abc123")
driver.find_element_by_name("password").send_keys("123xyz")
driver.find_element_by_name("submit").click()

# Storing cookies through Selenium
pickle.dump( driver.get_cookies() , open("cookies.pkl","wb"))
driver.quit()

# Passing cookies to Session
session = requests.session()  # or an existing session
with open('cookies.pkl', 'rb') as f:
    session.cookies.update(pickle.load(f))
search_requests = session.get('https://www.google.com/')
print(session.cookies)

Answer 2

Since you are replacing session.cookies (RequestsCookieJar) with a list which don't have those attributes, it won't work.

You can import those cookies one by one by using:

for c in your_cookies_list:
   initial_state.cookies.set(name=c['name'], value=c['value'])

I've tried loading the whole cookie but it seems like requests doesn't recognize those ones and returns:

TypeError: create_cookie() got unexpected keyword arguments: ['expiry', 'httpOnly']

requests accepts expires instead and HttpOnly comes nested within rest

Update:

We can also change the dict keys for expiry and httpOnly so that requests correctly load them instead of throwing an exception, by using dict.pop() which deletes an item from dict by the key and returns the value of deleted key so after we add a new key with deleted item value then unpack & pass them as kwargs:

for c in your_cookies_list:
    c['expires'] = c.pop('expiry')
    c['rest'] = {'HttpOnly': c.pop('httpOnly')}
    initial_state.cookies.set(**c)