Python - Requests pulling HTML instead of JSON

Question

I'm building a Python web scraper (personal use) and am running into some trouble retrieving a JSON file. I was able to find the request URL I need, but when I run my script (I'm using Requests) the URL returns HTML instead of the JSON shown in the Chrome Developer Tools console. Here's my current script:

import requests
import json

url = 'https://nytimes.wd5.myworkdayjobs.com/Video?clientRequestID=1f1a6071627946499b4b09fd0f668ef0'

r = requests.get(url)
print(r.text)

Completely new to Python, so any push in the right direction is greatly appreciated. Thanks!

Answer 1

Looks like that website returns the response depending on the accept headers provided by the request. So try:

import requests
import json

url = 'https://nytimes.wd5.myworkdayjobs.com/Video?clientRequestID=1f1a6071627946499b4b09fd0f668ef0'

r = requests.get(url, headers={'accept': 'application/json'})
print(r.json())

You can have a look at the full api for further reference: http://docs.python-requests.org/en/latest/api/.