Dynamically generated element -NoSuchElementException: Message: no such element: Unable to locate element?

Question

I am having trouble selecting a load more button on a Linkedin page. I receive this error in finding the xpath: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element

I suspect that the issue is that the button is not visible on the page at that time. So I have tried actions.move_to_element. However, the page scrolls just below the element, so that the element is no longer visible, and the same error subsequently occurs.

I have also tried move_to_element_with_offset, but this hasn't changed where the page scrolls to.

How can I scroll to the right location on the page such that I can successfully select the element?

My relevant code:

import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver

ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('C:\\Users\\Root\\Downloads\\chromedriver.exe')

driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)

username = driver.find_element_by_name('session_key')

username.send_keys(parameters.linkedin_username)
sleep(0.5)

password = driver.find_element_by_name('session_password')
password.send_keys(parameters.linkedin_password)
sleep(0.5)

sign_in_button = driver.find_element_by_xpath('//button[@class="btn__primary--large from__button--floating"]')
sign_in_button.click()

driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')

loadmore_skills=driver.find_element_by_xpath('//button[@class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')

actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()

Answer 1

After playing around with it, I seem to have figured out where the problem is stemming from. The error

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//button[@class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]"} (Session info: chrome=81.0.4044.113)

always correctly states the problem its encountering and as such it's not able to find the element. The possible causes of this include:

• Element not present at the time of execution
• Dynamically generated
• content Conflicting names

In your case, it was the second point. As the content that is displayed is loaded dynamically as you scroll down. So When it first loads your profile the skills sections aren't actually present in the DOM. So to solve this, you simply have to scroll to the section so that it gets applied in the DOM.

This line is the trick here. It will position it to the correct panel and thus loading and applying the data to the DOM.

driver.execute_script("window.scrollTo(0, 1800)")

Here's my code (Please change it as necessary)

from time import sleep

# import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait

ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('../chromedriver.exe')

driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)

username = driver.find_element_by_name('session_key')

username.send_keys('')
sleep(0.5)

password = driver.find_element_by_name('session_password')
password.send_keys('')
sleep(0.5)

sign_in_button = driver.find_element_by_xpath('//button[@class="btn__primary--large from__button--floating"]')
sign_in_button.click()


driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')

sleep(3)
# driver.execute_script("window.scrollTo(0, 1800)")
sleep(3)
loadmore_skills=driver.find_element_by_xpath('//button[@class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')


actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()

Output

Update

In concerns to your newer problem, you need to implement a continuous scroll method that would enable you to dynamically update the skills section. This requires a lot of change and should ideally be asked as a another question.

I have also found a simple solution by setting the scroll to the correct threshold. For y=3200 seems to work fine for all the profiles I've checked including yours, mine and few others.

driver.execute_script("window.scrollTo(0, 3200)")