I'm not able to use python requests session cookies in selenium

Question

I'm trying to open a requests session in a web browser it by the looks of it, it seems that using selenium is the most efficient/optimal way.

My code:

import requests
from selenium import webdriver
from time import sleep

s = requests.Session()
s.get('https://www.sotf.com/en/nike/man/footwear/nike--joyride--cc3--setter--sneakers--at6395.html?RwDet=true&articoli_ID=17911')

driver = webdriver.Safari()

driver.get("https://www.sotf.com/")

for cookie in s.cookies:
    driver.add_cookie({
        'name': cookie.name, 
        'value': cookie.value,
        'path': '/',
        'domain': cookie.domain,
    })

driver.refresh()
sleep(1000)

when printing s.cookies.get_dict() I get the following cookies:

{'__cfduid': 'dc81dd94c218523ce8161e4254d2652a01566815239', 'PHPSESSID': 'qhm7109shdrhu9uv3t38ani9df'}

The problem is that the browser doesn't isn't using these cookies, when checking the cookies inside of safari (using inspect element) __cfduid looks just like it should but for an unknown reason I see two PHPSESSID and the correct one has the Domain attribute set to .wwww.sotf.com instead of www.sotf.com:

Many thanks in advance.

Answer 1

The PHPSESSID cookie is stored twice because you open the page twice - first time you open the page with empty cookie jar, whereas the server sets the first insecure PHPSESSID cookie, then you copy the second one from the requests.Session. Clear the cookies once you land on the host; in the example below, I navigate to https://www.sotf.com/404 as 404 pages are usually faster to load, clear the default cookies, then copy cookies from requests' cookie jar:

import contextlib
import requests
from selenium import webdriver
from time import sleep


@contextlib.contextmanager
def init_driver():
    d = webdriver.Chrome()
    yield d
    d.quit()


if __name__ == '__main__':
    headers = {
        'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
        'accept-encoding': 'gzip, deflate, br',
        'accept-language': 'en-US,en;q=0.9,de;q=0.8',
        'sec-fetch-mode': 'navigate',
        'sec-fetch-site': 'none',
        'upgrade-insecure-requests': '1',
        'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36',
    }

    params = {
        'RwDet': 'true',
        'articoli_ID': '17911',
    }

    s = requests.Session()
    s.get('https://www.sotf.com/en/nike/man/footwear/nike--joyride--cc3--setter--sneakers--at6395.html', headers=headers, params=params)
    print('cookies in requests jar:')
    for c in s.cookies:
        print(c)


    with init_driver() as driver:
        # 404 pages are usually faster to load
        driver.get("https://www.sotf.com/404")
        driver.delete_all_cookies()

        for cookie in s.cookies:
            driver.add_cookie({
                'name': cookie.name,
                'value': cookie.value,
                'path': '/',
                'domain': cookie.domain,
            })

        driver.get("https://www.sotf.com/")
        print('cookies in selenium jar:')
        for c in driver.get_cookies():
            print(c)

Output:

cookies in requests jar:
<Cookie __cfduid=d54b8f9098af12dee16136e4dc641f74e1567012133 for .sotf.com/>
<Cookie PHPSESSID=mn28k5ta3ghfc77qb4nl23tga6 for www.sotf.com/>
cookies in selenium jar:
{'domain': 'www.sotf.com', 'expiry': 1598548157, 'httpOnly': False, 'name': 'cb-enabled', 'path': '/', 'secure': False, 'value': 'enabled'}
{'domain': 'www.sotf.com', 'httpOnly': False, 'name': 'PHPSESSID', 'path': '/', 'secure': True, 'value': 'mn28k5ta3ghfc77qb4nl23tga6'}
{'domain': 'sotf.com', 'httpOnly': False, 'name': '__cfduid', 'path': '/', 'secure': True, 'value': 'd54b8f9098af12dee16136e4dc641f74e1567012133'}