Extracting values from deeply nested JSON structures

Question

This is a structure I'm getting from elsewhere, that is, a list of deeply nested dictionaries:

{
    "foo_code": 404,
    "foo_rbody": {
        "query": {
            "info": {
                "acme_no": "444444",
                "road_runner": "123"
            },
            "error": "no_lunch",
            "message": "runner problem."
        }
    },
    "acme_no": "444444",
    "road_runner": "123",
    "xyzzy_code": 200,
    "xyzzy_rbody": {
        "api": {
            "items": [
                {
                    "desc": "OK",
                    "id": 198,
                    "acme_no": "789",
                    "road_runner": "123",
                    "params": {
                        "bicycle": "2wheel",
                        "willie": "hungry",
                        "height": "1",
                        "coyote_id": "1511111"
                    },
                    "activity": "TRAP",
                    "state": "active",
                    "status": 200,
                    "type": "chase"
                }
            ]
        }
    }
}

{
    "foo_code": 200,
    "foo_rbody": {
        "query": {
            "result": {
                "acme_no": "260060730303258",
                "road_runner": "123",
                "abyss": "26843545600"
            }
        }
    },
    "acme_no": "260060730303258",
    "road_runner": "123",
    "xyzzy_code": 200,
    "xyzzy_rbody": {
        "api": {
            "items": [
                {
                    "desc": "OK",
                    "id": 198,
                    "acme_no": "789",
                    "road_runner": "123",
                    "params": {
                        "bicycle": "2wheel",
                        "willie": "hungry",
                        "height": "1",
                        "coyote_id": "1511111"
                    },
                    "activity": "TRAP",
                    "state": "active",
                    "status": 200,
                    "type": "chase"
                }
            ]
        }
    }
}

Asking for different structures is out of question (legacy apis etc).

So I'm wondering if there's some clever way of extracting selected values from such a structure.

The candidates I was thinking of:

• flatten particular dictionaries, building composite keys, smth like:

  { "foo_rbody.query.info.acme_no": "444444", "foo_rbody.query.info.road_runner": "123", ... }

Pro: getting every value with one access and if predictable key is not there, it means that the structure was not there (as you might have noticed, dictionaries may have different structures depending on whether it was successful operation, error happened, etc).

Con: what to do with lists?

• Use some recursive function that would do successive key lookups, say by "foo_rbody", then by "query", "info", etc.

Any better candidates?

Answer 1

You can try this rather trivial function to access nested properties:

import re

def get_path(dct, path):
    for i, p in re.findall(r'(\d+)|(\w+)', path):
        dct = dct[p or int(i)]
    return dct

Usage:

value = get_path(data, "xyzzy_rbody.api.items[0].params.bicycle")