Selenium: Scroll to end of page in dynamically loading webpage

Question

I have a webpage that keeps loading new items when scrolling down the page until every item is loaded.

I'm working with Selenium in Java, and need to scroll down to the bottom of the page in order to load everything.

I have tried several different options, like scrolling to an element of the bottom of the page:

WebElement copyrightAtEndOfPage = webDriver.findElement(By.xpath("//a[@href='/utils/copyright.html']"));
((JavascriptExecutor) webDriver).executeScript("arguments[0].scrollIntoView();", copyrightAtEndOfPage);

This only scrolls down once though, and then the webpage keeps loading.

I also tried this approach, which also only scrolls down once, because it only takes the browser height into consideration.

Any help is highly appreciated.

Answer 1

I will provide you code in Python for this. I think it's easy to translate to Java:

def scroll_down(self):
    """A method for scrolling the page."""

    # Get scroll height.
    last_height = self.driver.execute_script("return document.body.scrollHeight")

    while True:

        # Scroll down to the bottom.
        self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

        # Wait to load the page.
        time.sleep(2)

        # Calculate new scroll height and compare with last scroll height.
        new_height = self.driver.execute_script("return document.body.scrollHeight")

        if new_height == last_height:

            break

        last_height = new_height

Hope it helps you!