Is there a way to multithread or batch REST API calls in Python?

Question

I've got a very long list of keys, and I am calling a REST API with each key to GET some metadata about it.

The API can only accept one key at a time, but I wondered if there was a way I could batch or multi-thread the calls from my side?

Answer 1

The other reply to this looks like ChatGPT so it should be ignored.

I did, however, use its code as a base to write a function that does what I want.

import requests
from concurrent.futures import ThreadPoolExecutor

API_ENDPOINT = 'https://api.example.com/metadata'

def get_metadata_for_key(key):
    url = f"{API_ENDPOINT}/{key}"
    response = requests.get(url)
    if response.status_code == 200:
        return response.json()
    else:
        return None

def get_save_metadata(keys, workers):
    results = {}
    batches = [keys[i : i + workers] for i in range(0, len(keys), workers)]

    with ThreadPoolExecutor(max_workers=workers) as executor:
        for batch in tqdm(batches):     #tqdm shows a progress bar
            futures = {key: executor.submit(get_metadata_for_key, key) for key in batch}
            futures_clean = {k: v.result() for k, v in futures.items() if v is not None}
            results.update({k: xmltodict.parse(v) for k, v in futures_clean.items()})
    
    return results