Screen scraping advice: Interactive graph

Question

I have recently followed some tutorials on how to use BeautifulSoup with Python and have learnt how to simply scrape text and urls from webpages. I am now trying to scrape data from the following link,

http://www.study.cam.ac.uk/undergraduate/apply/statistics/

There is an interactive graph generator at the bottom of the page and I would like to scrape all the data from it without having to spend many hours tediously handwriting down the values from all the possible graphs generated. I have tried to use my measly beginner techniques but it is not obvious to me where in the HTML the graph data is coming from - in addition the HTML seems to be dynamic depending on where my mouse is on the screen.

The Question: Is it possible to scrape this data using these tools and if so how?

Answer 1

Using browser developer tools, you can see that when you click on Show Graph button there is a POST request going to http://www.study.cam.ac.uk/undergraduate/apply/statistics/data.php. The result is a JSON object containing all of the data needed to build a graph.

Simulate this request in Python, for example, with requests module:

import requests

URL = "http://www.study.cam.ac.uk/undergraduate/apply/statistics/data.php"
HEADERS = {'X-Requested-With': 'XMLHttpRequest'}

data = {
    'when': 'year',
    'year': 2014,
    'applications': 'on',
    'offers': 'on',
    'acceptances': 'on',
    'groupby': 'college',
    'for-5-years-what': 'university'
}

response = requests.post(URL, data=data, headers=HEADERS)
print response.json()

No need for BeautifulSoup here. At least, from what I've understood from your question.