Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Efficient method to scroll though pages using Selenium

I'm currently hard coding a scrolling using Selenium, Firefox Driver in Python, is there anyway for it to scroll smoothly up and down the page? The page can grow 20 times bigger than normal as the images in the page loads.

I need to scroll up and down multiple times , not missing out any part of the page.

The current method isnt efficient and sometimes doesnt scroll to some part of the page. The following is my current code

 browser = webdriver.Firefox()
browser.get(link)

browser.execute_script("window.scrollTo(0, document.body.scrollHeight/3.4);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/3.5);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/3.7);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/3.8);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4.2);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4.3);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4.5);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4.7);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/4.9);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/5.2);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/5.1);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/5.8);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/3.7);")
        time.sleep(0.2)
        browser.execute_script("window.scrollTo(0, document.body.scrollHeight/50);")
        time.sleep(0.2)
like image 977
CodeGuru Avatar asked Jan 12 '23 23:01

CodeGuru


1 Answers

You can use this if using Selenium in Python...

scheight = .1
while scheight < 9.9:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight/%s);" % scheight)
    scheight += .01

Or something along those lines. You can add if brakes to capture data at specific scheight locations within the while. Just make the += or iteration whatever you like. I used .01 so I could visually see it scroll, otherwise you would need timeouts and it would appear choppy. Depends on what you need.

like image 146
Shane Avatar answered Jan 30 '23 22:01

Shane