Why Selenium webdriver with Python can't reach to a website

Question

I am trying to use selenium with chrome driver to connect to a website. But it couldn't be reached. Here is my code:

    from selenium import webdriver
    from selenium.webdriver.common.by import By
    
    CHROME_EXECUTABLE_PATH = "C://Program Files (x86)//Chrome Driver//chromedriver.exe"
    CHROME_OPTIONS = webdriver.ChromeOptions()
    CHROME_OPTIONS.add_argument("--disable-notifications")
    BASE_URL = "https://www.nordstrom.com/"
    driver = webdriver.Chrome(executable_path=CHROME_EXECUTABLE_PATH, options=CHROME_OPTIONS)
    # locators
    search_button_locator = "//a[@id='controls-keyword-search-popover']"
    search_box_locator = "//*[@id='keyword-search-input']"
    
    driver.get(BASE_URL)
    driver.find_element(By.XPATH, search_button_locator)
    driver.find_element(By.XPATH, search_box_locator).send_keys("Fave Slipper")

This code gives me some error:

E:\Python\Nordstrom.com\venv\Scripts\python.exe E:/Python/Nordstrom.com/pages/simple.py
Traceback (most recent call last):
  File "E:\Python\Nordstrom.com\pages\simple.py", line 14, in <module>
    driver.find_element(By.XPATH, search_button_locator)
  File "E:\Python\Nordstrom.com\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
    return self.execute(Command.FIND_ELEMENT, {
  File "E:\Python\Nordstrom.com\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "E:\Python\Nordstrom.com\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//a[@id='controls-keyword-search-popover']"}
  (Session info: chrome=94.0.4606.61)


Process finished with exit code 1

The page looks like this:

But the expected page should be looks like this:

How to access this website?

Answer 1

The error points out that it was unable to find the XPATH element, which is why it errored out.

The main causes for this can be either:

• the XPATH is wrong
• the element has not loaded yet on the page
• the site has detected your scraping attempt and blocked you

In this case it's a combination of the 2nd and 3rd options. Whenever you use a webdriver, it exposes javascript hooks that websites can detect. To hide your activity you should learn more on how device fingerprinting and either customize your script to hide itself or use a pre-made solution for it (such as PhantomJS).

Most likely you should also look into hiding your IP by using a proxy.