Scraping : Extract data from chart

Question

I found a way to download data from the chart page of the Wall Street Journal by looking at the network tab (in the dev tools panels) and reproducing the request that is created while refreshing the chart. It works as follow:

import requests
import json
import time
from datetime import datetime as dt
from urllib.parse import urlencode 

data = {
        "Step":"PT5M",
        "TimeFrame":"D1",
        "StartDate":int(dt(2019, 5, 1).timestamp()*1000),
        "EndDate":int(dt(2019, 5, 5).timestamp()*1000),
        "EntitlementToken":"57494d5ed7ad44af85bc59a51dd87c90",
        "IncludeMockTick":True,
        "FilterNullSlots":False,
        "FilterClosedPoints":True,
        "IncludeClosedSlots":False,
        "IncludeOfficialClose":True,
        "InjectOpen":False,
        "ShowPreMarket":False,
        "ShowAfterHours":False,
        "UseExtendedTimeFrame":True,
        "WantPriorClose":False,
        "IncludeCurrentQuotes":False,
        "ResetTodaysAfterHoursPercentChange":False,
        "Series":[{"Key":"STOCK/US/XNYS/ABB","Dialect":"Charting","Kind":"Ticker","SeriesId":"s1","DataTypes":["Last"],"Indicators":[{"Parameters":[{"Name":"Period","Value":"50"}],"Kind":"SimpleMovingAverage","SeriesId":"i2"},{"Parameters":[],"Kind":"Volume","SeriesId":"i3"}]}]
    }

data = {
    'json' : json.dumps(data)
}

data = urlencode(data)

headers = {
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'Dylan2010.EntitlementToken': '57494d5ed7ad44af85bc59a51dd87c90',
    'Origin': 'https://quotes.wsj.com',
    'Referer': 'https://quotes.wsj.com/ABB/advanced-chart',
    'Sec-Fetch-Mode': 'cors',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'
}

url = 'https://api.wsj.net/api/michelangelo/timeseries/history?' + data + '&ckey=57494d5ed7'
print(url)
r = requests.get(url, headers = headers)
r.text

This is great as its quite simple and works, however I can only retrieve minutes data that is maximum 25 days old or so.

On the other end, the morningstar charting seems to have much more minutes data available, and I would like to do the same thing with it : simply get the data from the website by looking up the javascript calls that are made in the background while updating the chart. But when I look at the network tab, I cannot see any call being made when changing the date range. I don't know much about javascript and would like to know what alternative mechanism they use to achieve that. (maybe async / fetch ?)

Does anyone know how it would be possible for me to see those calls ?

Answer 1

You will need to investigate if any of the other query string params are time based (or ticker dependant). I have replaced several params with a fixed value that still seems to work.

import requests

start_date = '20170814'
end_date = '20190102'

r = requests.get(f'https://quotespeed.morningstar.com/ra/uniqueChartData?instid=MSSAL&sdkver=2.39.0&CToken=1&productType=sdk&cdt=7&ed={end_date}&f=d&hasPreviousClose=true&ipoDates=19801212&pids=0P000000GY&sd={start_date}&tickers=126.1.AAPL&qs_wsid=27E31E614F74FC7D8828E941CAC2D319&tmpid=1&instid=MSSAL&_=1').json()

print(r)

---

Original params as observed with fiddler: