How to get large size JSON files using requests module in Python

Question

I am trying to save a large JSON file into a variable using requests module but only part of the JSON is making it into the variable when using the following:

r = requests.get(url)

r.json()

I see there are ways to save it in chunks when writing to a file is there a way to do this when writing to a variable?

Answer 1

According to the docs , using 'with' and setting stream to True should get you what you need.

with requests.get('http://httpbin.org/get', stream=True) as r:
    # Do things with the response here.