Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Getting selenium to work on pythonanywhere

My understanding is that pythonanywhere supports a headless Firefox browser but you need

from pyvirtualdisplay import Display

And so you can connect using

with Display():
    while True:
        try:
            driver = webdriver.Firefox()
            break
        except:
            time.sleep(3)

And I connect just fine. However, after I start using the driver with

with Display():
    while True:
        try:
            driver = webdriver.Firefox()
            break
        except:
            time.sleep(3)
    wb=load_workbook(r'/home/hoozits728/mutual_fund_tracker/Mutual_Fund_Tracker.xlsx')
    ws=wb.get_sheet_by_name('Tactical')

    for i in range(3, ws.max_row+1):
        if ws.cell(row=i,column=2).value is not None:
            driver.get('https://finance.yahoo.com/quote/' + ws.cell(row=i,column=2).value + '/performance?ltr=1')
            oneyear=driver.find_element_by_css_selector('#Col1-0-Performance-Proxy > section > div:nth-child(2) > div > div:nth-child(5) > span:nth-child(2)').text
            threeyear=driver.find_element_by_css_selector('#Col1-0-Performance-Proxy > section > div:nth-of-type(2) > div > div:nth-of-type(6) > span:nth-of-type(2)').text
            fiveyear=driver.find_element_by_css_selector('#Col1-0-Performance-Proxy > section > div:nth-of-type(2) > div > div:nth-of-type(7) > span:nth-of-type(2)').text
            ws.cell(row=i,column=10).value=oneyear
            ws.cell(row=i,column=11).value=threeyear
            ws.cell(row=i,column=12).value=fiveyear

           … and so on …

I get this error after just a little while

enter image description here

For what it's worth, this code works perfectly fine on my local machine. Also, I am a paying member, so there should be no whitelist issues.

like image 716
Matt Cremeens Avatar asked Sep 16 '18 11:09

Matt Cremeens


People also ask

Can you use Selenium on PythonAnywhere?

NOTE These instructions are for the system image fishnchips, glastonbury or haggis. If you have an older system image, you will need to update your system image to use these instructions. Also note that selenium will only work in tasks, web apps and consoles.

Can I run Selenium on Jupyter Notebook?

We've seen in past how to install and run Selenium with Python, in this guide we will try to do the same in Jupyter Notebook on WSL2. The instructions should be same for both WSL2 and Ubuntu.

How do you use Selenium for PIP?

To install the Selenium bindings in our system, run the command: pip install selenium. As this is done, a folder called Selenium should get created within the Python folder. To update the existing version of Selenium, run the command: pip install –U selenium.


1 Answers

It has recently come to my understanding that yahoo has blocked pythonanywhere from running any web scraping scripts. I assume this is true for all AWS servers and those who use them, but I am not 100% certain of this. I hope this helps anyone who comes across this question.

https://www.pythonanywhere.com/forums/topic/5724/#id_post_52307

like image 130
Matt Cremeens Avatar answered Oct 08 '22 01:10

Matt Cremeens