How to interact with a javascript webpage using Python?

Question

I am a complete newbie to web scraping; I have this small project of scraping some data from COCA but I don't even know where to start. It seems that this webpage is built using some Javascript and I wonder if there is some package that enables me to interact with it?

Here is some tasks that I want my program to do:

1. log in using one's account;
2. Choose a tab (e.g. search, chart, etc, please see COCA);
3. type in the word you want to search in the textbook;
4. scrape the search results.

Any suggestions would be greatly appreciated.

PS: Ideally everything should work at backstage (won't open the browser).

Answer 1

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(800, 600))
display.start()

browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()

display.stop()

pyvirtualdisplay in headless mode Display(visible=0) requires Xvbf, that is a feature of Linux. Read more here on Xvbf usage.

Answer 2

As some people have told you, you can use selenium. I recommend you to enter in the developers tools of your browser and follow the network requests that make the site, depending of the behavior of the page maybe you can do it with the python module request to simulate the request that you saw that was making the site, personally i think that it is simpler. If you can't emulate the request then use selenium.