My understanding is that pythonanywhere supports a headless Firefox browser but you need
from pyvirtualdisplay import Display
And so you can connect using
with Display():
while True:
try:
driver = webdriver.Firefox()
break
except:
time.sleep(3)
And I connect just fine. However, after I start using the driver with
with Display():
while True:
try:
driver = webdriver.Firefox()
break
except:
time.sleep(3)
wb=load_workbook(r'/home/hoozits728/mutual_fund_tracker/Mutual_Fund_Tracker.xlsx')
ws=wb.get_sheet_by_name('Tactical')
for i in range(3, ws.max_row+1):
if ws.cell(row=i,column=2).value is not None:
driver.get('https://finance.yahoo.com/quote/' + ws.cell(row=i,column=2).value + '/performance?ltr=1')
oneyear=driver.find_element_by_css_selector('#Col1-0-Performance-Proxy > section > div:nth-child(2) > div > div:nth-child(5) > span:nth-child(2)').text
threeyear=driver.find_element_by_css_selector('#Col1-0-Performance-Proxy > section > div:nth-of-type(2) > div > div:nth-of-type(6) > span:nth-of-type(2)').text
fiveyear=driver.find_element_by_css_selector('#Col1-0-Performance-Proxy > section > div:nth-of-type(2) > div > div:nth-of-type(7) > span:nth-of-type(2)').text
ws.cell(row=i,column=10).value=oneyear
ws.cell(row=i,column=11).value=threeyear
ws.cell(row=i,column=12).value=fiveyear
… and so on …
I get this error after just a little while
For what it's worth, this code works perfectly fine on my local machine. Also, I am a paying member, so there should be no whitelist issues.
NOTE These instructions are for the system image fishnchips, glastonbury or haggis. If you have an older system image, you will need to update your system image to use these instructions. Also note that selenium will only work in tasks, web apps and consoles.
We've seen in past how to install and run Selenium with Python, in this guide we will try to do the same in Jupyter Notebook on WSL2. The instructions should be same for both WSL2 and Ubuntu.
To install the Selenium bindings in our system, run the command: pip install selenium. As this is done, a folder called Selenium should get created within the Python folder. To update the existing version of Selenium, run the command: pip install –U selenium.
It has recently come to my understanding that yahoo has blocked pythonanywhere from running any web scraping scripts. I assume this is true for all AWS servers and those who use them, but I am not 100% certain of this. I hope this helps anyone who comes across this question.
https://www.pythonanywhere.com/forums/topic/5724/#id_post_52307
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With