Selenium: driver.get_cookies() returns incomplete list of cookies

Question

The cookies returned are as follows:

[
    {
        "domain": "www.carid.com",
        "name": "uxatc",
        "value": "%13%18%07%13%0D%07%13J%07%14J%07%0DJ%07%13%14%07%13I%07%12%13%0F%09%0E%1E%07%13%1F%1D%1A%17%08%1E%07%14%0F%1D%1A%17%08%1E%07%08%17%0F%09%0E%1E%07%16%16%07%16%1F%07%0F%08%07%0F%16%07%10%1F%07%15%16%07%0D%08%0D%12%08%12%19%17%1E%07%0C%10%07%1F%1F%07%1F%10%07%0C%1F%07%15%0C%1D%1A%17%08%1E%07%1F%19%07%03%02%07%08%18%07%0FJJNMMINOJLJCLK%07%0FI%07",
        "path": "/",
        "httpOnly": false,
        "secure": false
    },
    {
        "domain": ".carid.com",
        "name": "xid",
        "value": "21f70e8bba820dcaf620307b1717f90c",
        "path": "/",
        "httpOnly": true,
        "secure": false
    },
    {
        "domain": ".carid.com",
        "name": "xidRes",
        "value": "21f70e8bba820dcaf620307b1717f90c",
        "expiry": 1567463776,
        "path": "/",
        "httpOnly": true,
        "secure": false
    },
    {
        "domain": "www.carid.com",
        "name": "store_language",
        "value": "US",
        "expiry": 1597790176,
        "path": "/",
        "httpOnly": false,
        "secure": true
    },
    {
        "domain": "www.carid.com",
        "name": "uxat",
        "value": "%13%18K%07%0DJ%07%13J%1D%1A%17%08%1E%07%14J%1D%1A%17%08%1E%07%13%1F%1D%1A%17%08%1E%07%14%0F%1D%1A%17%08%1E%07%08%17%0F%09%0E%1E%07%16%16%07%16%1F%07%0F%08%07%0F%16%07%10%1F%07%15%16%07%0D%08%0D%12%08%12%19%17%1E%07%0C%10%07%1F%1F%07%1F%10%07%0C%1F%07%15%0C%1D%1A%17%08%1E%07%1F%19%07%13IK%07",
        "path": "/",
        "httpOnly": false,
        "secure": false
    },
    {
        "domain": "www.carid.com",
        "name": "uxid2",
        "value": "16fc88Ve3eb83cfbccGc5",
        "path": "/",
        "httpOnly": false,
        "secure": false
    },
    {
        "domain": "www.carid.com",
        "name": "uxid",
        "value": "8a58hVe3eb83cfc13Ym8",
        "path": "/",
        "httpOnly": false,
        "secure": false
    }
]

There are 7 cookies, but the browser has 17 cookies:

Why are there missing cookies? Is there a way to get all the cookies?

Answer 1

Possibly, because you didn't wait for the page to fully load before getting the cookies. I checked the site and observe the cookies status. Before the page fully loads on car selecting divs, there are less cookies than after the page is fully loaded.

Even if the driver thinks that the page is fully loaded, maybe it's not. It might run some background scripts. It's best to check for it before you want to do anything.

Working example:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
opt = webdriver.FirefoxOptions()
# opt.headless = True
driver = webdriver.Firefox(options=opt)
driver.delete_all_cookies()
driver.get("https://carid.com")
cookies_before = driver.get_cookies()
WebDriverWait(driver, 15).until(
        EC.visibility_of_element_located(
                (By.XPATH, '//div[contains(@class, "select-vehicle-button")]')
        ))
print("Loaded!")
cookies_after = driver.get_cookies()
print(bool(cookies_before == cookies_after))
print(f"Cookies before: {len(cookies_before)}.")
print(f"Cookies after: {len(cookies_after)}.")
print(cookies_after)
driver.quit()

Output when before & after are different (only sometimes):

Loaded!
False
Cookies before: 5.
Cookies after: 17.

---

Notes: careful if you're running this in headless. Chrome headless will not work on this site, Firefox headless will.