Storing response as json in Python requests-cache

Question

I'm using requests-cache to cache http responses in human-readable format. I've patched requests using the filesystem backend, and the the serializer to json, like so:

import requests_cache
requests_cache.install_cache('example_cache', backend='filesystem', serializer='json')

The responses do get cached as json, but the response's body is encoded (I guess using the cattrs library, as described here).

Is there a way to make requests-cache save responses as-is?

Answer 1

What you want to do makes sense, but it's a bit more complicated than it appears. The response files you see are representations of requests.Response objects. Response._content contains the original bytes received from the server. The wrapper methods and properties like Response.json() and Response.text will then attempt to decode that content. For a Response object to work correctly, it needs to have the original binary response body.

When requests-cache serializes that response as JSON, the binary content is encoded in Base85. That's why you're seeing encoded bytes instead of JSON there. To have everything including the response body saved in JSON, there are a couple options:

Option 1

Make a custom serializer. If you wanted to be able to modify response content and have those changes reflected in responses returned by requests-cache, this would probably be the best way to do it.

This may be become a bit convoluted, because you would have to:

1. Handle response content that isn't valid JSON, and save as encoded bytes instead
2. During deserialization, if the content was saved as JSON, convert it back into bytes to recreate the original Response object

It's doable, though. I could try to come up with an example later, if needed.

Option 2

Make a custom backend. It could extend FileCache and FileDict, and copy valid JSON content to a separate file. Here is a working example:

import json
from os.path import splitext

from requests import Response
from requests_cache import CachedSession, FileCache, FileDict


class JSONFileCache(FileCache):
    """Filesystem backend that copies JSON-formatted response content into a separate file
    alongside the main response file
    """
    def __init__(self, cache_name, **kwargs):
        super().__init__(cache_name, **kwargs)
        self.responses = JSONFileDict(cache_name, **kwargs)



class JSONFileDict(FileDict):
    def __setitem__(self, key: str, value: Response):
        super().__setitem__(key, value)
        response_path = splitext(self._path(key))[0]
        json_path = f'{response_path}_content.json'

        # Will handle errors and skip writing if content can't be decoded as JSON
        with self._try_io(ignore_errors=True):
            content = json.dumps(value.json(), indent=2)
            with open(json_path, mode='w') as f:
                f.write(content)

Usage example:

custom_backend = JSONFileCache('example_cache', serializer='json')
session = CachedSession(backend=custom_backend)
session.get('https://httpbin.org/get')

After making a request, you will see a pair of files like:

example_cache/680f2a52944ee079.json
example_cache/680f2a52944ee079_content.json

That may not be exactly what you want, but it's the easiest option if you only need to read the response content and don't need to modify it.