scrolling down in selenium while more Data loads- Lazy loading

Question

Hi I have written a code to scroll down a page using the following :

JavascriptExecutor js = (JavascriptExecutor) driver;
js.executeScript("window.scrollTo(0, document.body.scrollHeight)");

However, it scrolls only once, while more data loads on my page. The data is enclosed inside a div tag.

When it scrolls down, The page shows loading and loads more data (Lazy loading). How do i implement scrolling for lazy loading in it?

Answer 1

Here example how you can scroll until size of data elements not change. To get loadingLocator open chrome devtools, scroll and press F8, it will pause and you'll able to get selector.

WebDriverWait wait = new WebDriverWait(driver, 5);
JavascriptExecutor js = (JavascriptExecutor) driver;

int dataSize = driver.findElements(dataLocator).size();
while (true){
    js.executeScript("window.scrollTo(0, document.body.scrollHeight)");
    wait.ignoring(NoSuchElementException.class)
            .until(ExpectedConditions.invisibilityOfElementLocated(loadingLocator));

    if (driver.findElements(dataLocator).size()== dataSize)
        break;
    dataSize = driver.findElements(dataLocator).size();
}

Also you can implement your own wait like below:

By dataLocator = By.cssSelector(".save_ride_container");
WebDriverWait wait = new WebDriverWait(driver, 5, 500);
JavascriptExecutor js = (JavascriptExecutor) driver;

AtomicInteger dataSize = new AtomicInteger(driver.findElements(dataLocator).size());
boolean complete = false;
while (!complete){
    js.executeScript("window.scrollTo(0, document.body.scrollHeight)");
    complete = wait.ignoring(TimeoutException.class)
            .until(d -> {
                int size = driver.findElements(dataLocator).size();
                if (size== dataSize.get())
                    return true;
                dataSize.set(size);
                return false;
            });
}

List<WebElement> passengers = driver.findElements(dataLocator);
passengers.forEach(p -> System.out.println(p.findElement(By.xpath("./div[1]/div[3]/p[5]/strong"))));