Handling "Accept Cookies" popup with Selenium in Python

Question

I've been trying to scrape some information of this real estate website with selenium. However when I access the website I need to accept cookies to continue. This only happens when the bot accesses the website, not when I do it manually. When I try to find the corresponding element either by xpath or id, as I find it when I inspect the page manually I get following error.

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="uc-btn-accept-banner"]"}

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC



PATH = "/usr/bin/chromedriver"
driver = webdriver.Chrome(PATH)

driver.get("https://www.immoweb.be/en/search/house/for-sale?countries=BE&page=1&orderBy=relevance")
driver.find_element_by_xpath('//*[@id="uc-btn-accept-banner"]').click()

Does anyone know how to get around this? Why can't I find the element?

Below is an image of the accept cookies popup.

This is the HTML corresponding to the button "Keep browsing". The XPATH is as above.

<button aria-label="Keep browsing" id="uc-btn-accept-banner" class="uc-btn-new  uc-btn-accept">Keep browsing
          <span id="uc-optin-timer-display"></span></button>

Answer 1

You were very close!

If you open your page in a new browser you'll note the page fully loads, then, a moment later your popup appears.

The default wait strategy in selenium is just that the page is loaded. That draw delay between page loaded and display appearing is causing your scripts to fail.

You have two good synchronisation options.

1/ Include an implicit wait for your driver. This is done once per script and affects all objects. This waits 10 seconds before throwing any error, or continues when it's ready:

PATH = "/usr/bin/chromedriver"
driver = webdriver.Chrome(PATH)
driver.implicitly_wait(10)

driver.get("https://www.immoweb.be/en/search/house/for-sale?countries=BE&page=1&orderBy=relevance")
driver.find_element_by_xpath('//*[@id="uc-btn-accept-banner"]').click()

2/ Do a explicit wait on your object only:

PATH = "/usr/bin/chromedriver"
driver = webdriver.Chrome(PATH)

driver.get("https://www.immoweb.be/en/search/house/for-sale?countries=BE&page=1&orderBy=relevance")
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH,'//*[@id="uc-btn-accept-banner"]'))).click()

More info on the wait strategies is here