python selenium: does not wait until page is loaded after a click() command

Question

does someone know how to wait until the page is loaded? I tried all possible variants I found on the web but is simply does not work.

I need to wait after I trigger a click() command, there is some internal scripting on the web server which fools checks such as (I exclude the code to import required modules and use standard naming conventions):

WebDriverWait(browser, 10).until(lambda d: d.find_element_by_id(the_id))

or

browser.implicitly_wait(10) 

or

elem2=wait.until(EC.presence_of_element_located((By.ID,the_id)))

all the above checks do not work, in the sense that they return True even if the page is still loading. This causes text which I am reading to be incomplete since the page is not fully loaded after a click() command. It seems that in python the command clickAndWait is not available (but again, it probably would not solve the problem as the other tests also fail). Ideally there would be a command which waits until the whole web page is loaded (regardless of a particular element of the page).

I managed to solve the problem by manually inserting a time.sleep(5) in a loop, but this is sub-optimal since it might slow down the whole process. If possible, better to wait only the strictly required time.

Thx

Answer 1

Fundamentally if the page is being generated by JavaScript then there is no way to tell when it has "finished" loading.

However. I typically try to retrieve a page element, catch the exception and keep on trying until a timeout is reached. I might also check that the element is visible not merely present.

There must be some DOM element whose visibility you can use to test that a page has "finished" loading. I typically have a wait_for_element_visibility, and wait_for_element_presence functions in my test cases.

def wait_for_visibility(self, selector, timeout_seconds=10):
    retries = timeout_seconds
    while retries:
        try:
            element = self.get_via_css(selector)
            if element.is_displayed():
                return element
        except (exceptions.NoSuchElementException,
                exceptions.StaleElementReferenceException):
            if retries <= 0:
                raise
            else:
                pass

        retries = retries - 1
        time.sleep(pause_interval)
    raise exceptions.ElementNotVisibleException(
        "Element %s not visible despite waiting for %s seconds" % (
            selector, timeout_seconds)
    )

get_via_css is one of my own functions but I hope it's fairly clear what it is doing.

Answer 2

Here's my version that I have used:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def waitForLoad(inputXPath): 
    Wait = WebDriverWait(browser, PATIENCE_TIME)       
    Wait.until(EC.presence_of_element_located((By.XPATH, inputXPath)))

The top line in the function is a generic wait. The second one is an expected conditions where it is trying to find an element that you know exists on the page. You can change it so instead of Xpath it does CSS_SELECTOR, NAME, ID, CLASS, etc. there are many others, but you'd have to look up the expected conditions documentation. Hope that helped, it helped me when I figured it out.