How to preload cookies before first request with Python3, Selenium Chrome WebDriver?

Question

Is it possible to add cookies using add_cookie() for a domain, say stackoverflow.com in Selenium Chrome WebDriver before doing an actual request using get() to a page on domain stackoverflow.com ?

When trying:

driver.webdriver.add_cookie({'name' : 'testcookie', 'value' : 'testvalue', 'domain' : 'stackoverflow.com'})
driver.webdriver.get('https://stackoverflow.com/')

I get "You may only set cookies for the current domain".

I want to say I saw and tried some solutions of avoiding the issue instead of solving it, such as visiting a 404 page on the domain beforehand to create the "domain slot" in Selenium before adding cookies to it, but while these solution allow adding the cookies they still require making one extra request and making contact with the site while not having any cookies set.

This is an issue when dealing with CAPTCHA systems and some very specific WAFs which frown upon seeing, in succession, a request with no cookies then another request with cookies that should only be had say after going through the login process.

Answer 1

Starting Chrome 64 we now have access to Chrome DevTools Protocol v1.3 which allows setting cookies to any domain through the method Network.setCookie, thus eliminating the need for an extra get call to set up the domain beforehand.

e.g. Python3

import os.path
import pickle
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

def save_cookies():
    print("Saving cookies in " + selenium_cookie_file)
    pickle.dump(driver.get_cookies() , open(selenium_cookie_file,"wb"))

def load_cookies():
    if os.path.exists(selenium_cookie_file) and os.path.isfile(selenium_cookie_file):
        print("Loading cookies from " + selenium_cookie_file)
        cookies = pickle.load(open(selenium_cookie_file, "rb"))

        # Enables network tracking so we may use Network.setCookie method
        driver.execute_cdp_cmd('Network.enable', {})

        # Iterate through pickle dict and add all the cookies
        for cookie in cookies:
            # Fix issue Chrome exports 'expiry' key but expects 'expire' on import
            if 'expiry' in cookie:
                cookie['expires'] = cookie['expiry']
                del cookie['expiry']

            # Replace domain 'apple.com' with 'microsoft.com' cookies
            cookie['domain'] = cookie['domain'].replace('apple.com', 'microsoft.com')

            # Set the actual cookie
            driver.execute_cdp_cmd('Network.setCookie', cookie)

        # Disable network tracking
        driver.execute_cdp_cmd('Network.disable', {})
        return 1

    print("Cookie file " + selenium_cookie_file + " does not exist.")
    return 0

def pretty_print(pdict):
    for p in pdict:
        print(str(p))
    print('',end = "\n\n")


# Minimal settings
selenium_cookie_file = '/home/selenium/test.txt'

browser_options = Options()
browser_options.add_argument("--headless")

# Open a driver, get a page, save cookies
driver = webdriver.Chrome(chrome_options=browser_options)
driver.get('https://apple.com/')
save_cookies()
pretty_print(driver.get_cookies())


# Rewrite driver with a new one, load and set cookies before any requests
driver = webdriver.Chrome(chrome_options=browser_options)
load_cookies()
driver.get('https://microsoft.com')
pretty_print(driver.get_cookies())

You will see above that we get cookies from domain 'apple.com' with one request and have them loaded and set for the second request to domain 'microsoft.com' before actually making the request to 'microsoft.com'.

The code is tested and works as long as you set the variable selenium_cookie_file to a valid writable file path.