I am using selenium to scrape an infinite scrolling page.
I am trying to use this code:
import time
import pandas as np
import numpy as np
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
browser = webdriver.Chrome()
url = 'https://twitter.com/search?f=tweets&q=csubwaystats%20since%3A2018-05-28%20until%3A2018-08-28'
browser.get(url)
time.sleep(1)
SCROLL_PAUSE_TIME = 0.5
# Get scroll height
last_height = webdriver.execute_script("return document.body.scrollHeight")
while True:
# Scroll down to bottom
webdriver.execute_script("window.scrollTo(0,document.body.scrollHeight);")
# Wait to load page
time.sleep(SCROLL_PAUSE_TIME)
# Calculate new scroll height and compare with last scroll height
new_height = webdriver.execute_script("return document.body.scrollHeight")
if new_height == last_height:
break
last_height = new_height
I obtained this code from multiple sources, the most recent being:
How can I scroll a web page using selenium webdriver in python?
I updated it to include "webdriver" instead of "driver" because I import selenium as webdriver. It doesn't work otherwise.
My issue is that when I run the code I get:
AttributeError: module 'selenium.webdriver' has no attribute 'execute_script'
I don't really understand what this means and how to fix it? I haven't been able to find information on this.
I am new to python and so am probably missing something obvious but any advice would be appreciated.
webdriver
is the name of the module, not your instance of it. In fact, you assigned the instance you created to the name browser
with this line: browser = webdriver.Chrome()
so instead of calling webdriver.execute_script()
(which will give you an AttributeError
), you must call it using your instance, like this: browser.execute_script()
.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With