Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Scrolling down a page with Selenium Webdriver

I have a dynamic page that loads products when the user scrolls down a page. I want to get the total number of products rendered on the display page. Currently I am using the following code to get to the bottom until all the products are being displayed.

elems = WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x")))
print len(elems)
a = len(elems)
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(4)
elem1 = WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x")))
b = len(elem1)
while b > a:
    self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(4)
    elem1 = WebDriverWait(self.driver, 30).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "x")))
    a = b
    b = len(elem1)
print b

This is working nicely, but I want to know whether there is any better option of doing this?

like image 890
Saheb Avatar asked Feb 13 '14 11:02

Saheb


People also ask

How do I scroll down and scroll in Selenium?

New Selenium IDE Selenium can execute JavaScript commands with the help of the executeScript method. To scroll down vertically in a page we have to use the JavaScript command window. scrollBy. Then pass the pixel values to be traversed along the x and y axis for horizontal and vertical scroll respectively.

How do you scroll down a page?

Scroll one page at a time in all major browsers including Microsoft Internet Explorer and Mozilla Firefox by pressing the Spacebar key. Move back up the page by pressing Shift + Spacebar or the Home key on the keyboard.


1 Answers

You can perform this action easily using this line of code

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

And if you want to scroll down for ever you should try this.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

driver = webdriver.Firefox()
driver.get("https://twitter.com/BarackObama")

while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(3)

I am not sure about time.sleep(x value) cause loading data my take longer .. or less .. for more information please check the official Doc page

have fun :)

like image 99
Ayyoub Avatar answered Sep 24 '22 05:09

Ayyoub