Python to retrieve multiple pages of data from API with GET

Question

I am trying to use Python 3 requests.get to retrieve the data from this page, using its API. I am interested in retrieving data from all the pages using the API.

Here is my attempt so far

data = 'https://api.safecast.org/en-US/measurements'
data = requests.get(url)

My problem is the following - when I check the length of data using

len(data.json())

it gives me 25. This is because there are 25 records per page and it is only returning page number 1. I need to retrieve data from all pages, not just page 1.

According to the API, there are some parameters that can be specified in the query in order to filter the search. But, I do not know how to specify the page number in the query.

I looked through these 2 SO posts (1, 2), but I could not find something relevant to my problem.

Based on this post, I tried

print(data.links)

but this just gave {}

Question

Is there a way to collect data from all pages at once, using the API? Also, how do I determine the number of pages programmatically?

Answer 1

Hi As there are total 4458708 pages , you can add for loop and get json of each page. Check below code

import requests
for page in range(1,4458709):
    url = 'https://api.safecast.org/en-US/measurements.json?page=%s'%page
    data = requests.get(url)
    print data.json()