no such element: Unable to locate element using chromedriver and Selenium in production environment

Question

I have a problem with selenium chromedriver which I cannot figure out what's causing it. Some weeks ago everything was working OK, and suddenly this error started to show up. The problem is coming from the following function.

 def login_(browser):
    try:
        browser.get("some_url")
        # user credentials
        user = browser.find_element_by_xpath('//*[@id="username"]')
        user.send_keys(config('user'))
        password = browser.find_element_by_xpath('//*[@id="password"]')
        password.send_keys(config('pass'))
        login = browser.find_element_by_xpath('/html/body/div[1]/div/button')
        login.send_keys("\n")
        time.sleep(1)
        sidebar = browser.find_element_by_xpath('//*[@id="sidebar"]/ul/li[1]/a')
        sidebar.send_keys("\n")
        app_submit = browser.find_element_by_xpath('//*[@id="sidebar"]/ul/li[1]/ul/li[1]/a')
        app_submit.send_keys("\n")
    except TimeoutException or NoSuchElementException:
        raise LoginException

This function works with no problem in the development environment (macOS 10.11), but throws the following error in the production environment:

Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="sidebar"]/ul/li[1]/a"}
(Session info: headless chrome=67.0.3396.79)
(Driver info: chromedriver=2.40.565383 (76257d1ab79276b2d53ee97XXX),platform=Linux 4.4.0-116-generic x86_64)

I already updated both Chrome and chromedriver (v67 & 2.40, respectively) in each environment. I also gave it more time.sleep(15). But the problem persists. My latest guess is that maybe the initialization of the webdriver is not working properly:

def initiate_webdriver():
   option = webdriver.ChromeOptions()
   option.binary_location = config('GOOGLE_CHROME_BIN')
   option.add_argument('--disable-gpu')
   option.add_argument('window-size=1600,900')
   option.add_argument('--no-sandbox')
   if not config('DEBUG', cast=bool):
       display = Display(visible=0, size=(1600, 900))
       display.start()
       option.add_argument("--headless")
   else:
       option.add_argument("--incognito")
   return webdriver.Chrome(executable_path=config('CHROMEDRIVER_PATH'), chrome_options=option)

Because, if the Display is not working, then there may not be the mentioned sidebar but some other button.

So my questions are: does anybody have had a similar issue? Is there a way to know what is the page showing at the time the driver is looking for such an element?

Answer 1

You need to wait until the element is visible or else you will get this error. Try something like this:

from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support.expected_conditions import visibility_of_element_located
from selenium.webdriver.common.by import By
TIMEOUT = 5

...
xpath = '//*[@id="sidebar"]/ul/li[1]/a'
WebDriverWait(self.selenium, TIMEOUT).until(visibility_of_element_located((By.XPATH, xpath)))
browser.find_element_by_xpath(xpath)
...

Answer 2

Whenever I encounter strange issues in Selenium like this, I prefer retrying to find the particular element which is causing intermittent troubles. One way is to wrap it around a try-except block:

try:
   sidebar = browser.find_element_by_xpath('//*[@id="sidebar"]/ul/li[1]/a')
except NoSuchElementException:
   time.sleep(10)
   print("Unable to find element in first time, trying it again")
   sidebar = browser.find_element_by_xpath('//*[@id="sidebar"]/ul/li[1]/a')

You could also put the try code in a loop with a suitable count variable to make the automation code work. (Check this). In my experience with JAVA, this idea has resolved multiple issues.

Answer 3

A couple of things as per the login_(browser) method:

• As you have identified the Login button through:

  login = browser.find_element_by_xpath('/html/body/div[1]/div/button')
  

  I would suggest rather invoking send_keys("\n") take help of the onclick() event through login.click() to mock the clicking of Login button as follows:

  login = browser.find_element_by_xpath('/html/body/div[1]/div/button')
  login.click()
  

• Next when you identify the sidebar induce WebDriverWait for the element to be clickable as follows:

  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="sidebar"]/ul/li[1]/a'))).click()
  

• As you mentioned your code code block works perfect in macOS 10.11 environment but throws the following error in the production environment (Linux) it is highly possible that different browsers renders the HTML DOM differently in different OS architecture. So instead of absolute xpath you must use relative xpath as follows:

  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[@attribute='value']"))).click()
  

A couple of things as per the initiate_webdriver() method:

• As per Getting Started with Headless Chrome the argument --disable-gpu is applicable only for Windows but not a valid configuration for Linux OS. So need o remove:

  option.add_argument('--disable-gpu')
  

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC