Python, Selenium : 'Element is no longer attached to the DOM'

Question

I am scraping a website, www.lipperleaders.com. I want to extract the funds detail of Singapore. I have successfully implemented drop-down selection and extracted the content of the first page appearing after submission of the options. But when I try to go to next pages (by making the code to click next button) I am getting error 'Element is no longer attached to the DOM'.

My code is about 100 lines but I can give a general idea of the flow of execution of my code:

...                    # creating driver object and all the imports
def main():
    ...
    result = find_elements_by_tag_name('span')  
    ...
    driver.find_element_by_id("ctl00_ContentPlaceHolder1_ucDataPager_btnNext").click()
    main()
main()

This code works fine for the 1st page but when main() is called again after clicking of the next button. Before this recursive method, I also tried putting this inside a loop, then also same error.

And if I write the same code like:

# some code
result = find_elements_by_tag_name('span')  
driver.find_element_by_id("ctl00_ContentPlaceHolder1_ucDataPager_btnNext").click()
# some code
driver.find_element_by_id("ctl00_ContentPlaceHolder1_ucDataPager_btnNext").click()
.
.

This code works fine w/o any error the next page loads and executes the code written after that. But I cannot write the same driver.find_element_by_id().click() for 500 pages, even I will have to repeat the rest of the code associated with each page. That's why I am trying for loop or recursion, but its not working for me.

Please let me know what is the problem with my approach.

Answer 1

The problem is that the element is being detached by some javascript. So you should make the driver wait for the element: This is done by setting implicitly_wait, see:

from selenium import webdriver

ff = webdriver.Firefox()
ff.implicitly_wait(10) # seconds
...
myDynamicElement = ff.find_element_by_id("myDynamicElement")

from http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp#implicit-waits