Selenium error "Element is no longer attached to the DOM" while scraping data

Question

        for i in driver.find_elements_by_class_name("endorse-count"):
            try:
                i.click()
            except:
                continue

            elem = WebDriverWait(driver, 100).until(EC.presence_of_element_located((By.CLASS_NAME, "dialog-window")))
            src = elem.get_attribute("innerHTML")

            add_skill(name, src)

            WebDriverWait(driver, timeout=10)

I'm getting the following error while running the above code -

selenium.common.exceptions.StaleElementReferenceException: Message: u'Element is no longer attached to the DOM' ; Stacktrace: 
    at fxdriver.cache.getElementAt (resource://fxdriver/modules/web_element_cache.js:7646)

for line -

src = elem.get_attribute("innerHTML")

I'm running this code on LinkedIn user profile page, after logging in.

I tried putting the following line of code after "i.click()" -

driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);

But then I see that function "add_skill(name, src)" is not called and none of the code after driver.manage() is called, though for loop and further i.click() work fine.

Answer 1

Selenium is trying to complete actions (such as clicking a button or link) before verifying that the target element has rendered on the page. Selenium can be more patient, but you have to explicitly ask him to be.

For example, if you are testing something that makes an AJAX request, you can try something like this (in Ruby):

# timeout is in seconds
def wait_for_ajax(timeout=x)
 time_limit, interval = (Time.now + timeout), 0.5
 loop do
   break if @driver.execute_script "return jQuery.active == 0"
   sleep interval
   raise "Wait for AJAX timed out after waiting for #{timeout} seconds" if Time.now > time_limit
 end
end

To ensure your tests are fully comprehensive, always make Selenium waits for elements to load before running a task.

Answer 2

I had faced a similar issue and tried refreshing the page before finding that element, and it worked...

driver.navigate().refresh();

Though I couldnt reason out how this worked.

If this works for you as well, please let me know. I just want to learn more about this exception.

you can refer this page to learn about a similar issue