Dealing with optional python dictionary fields

Question

I'm dealing with JSON data which I load into Python dictionaries. A lot of these have optional fields, which then may contain dictionaries, that kind of stuff.

dictionary1 = 
{"required": {"value1": "one", "value2": "two"},
"optional": {"value1": "one"}}

dictionary2 = 
{"required": {"value1": "one", "value2": "two"}}

If I do this,

dictionary1.get("required").get("value1")

this works, obviously, because the field "required" is always present.

However, when I use the same line on dictionary2 (to get the optional field), this will produce an AttributeError

dictionary2.get("optional").get("value1")
AttributeError: 'NoneType' object has no attribute 'get'

which makes sense, because the first .get() will return None, and the second .get() cannot call .get() on the None object.

I can solve this by giving default values in case the optional field is missing, but this will be annoying the more complex the data gets, so I'm calling this a "naive fix":

dictionary2.get("optional", {}).get("value1", " ")

So the first .get() will return an empty dictionary {}, on which the second .get() can be called, and since it obviously contains nothing, it will return the empty string, as defined per the second default.

This will no longer produce errors, but I was wondering if there is a better solution for this - especially for more complex cases (value1 containing an array or another dictionary, etc....)

I could also fix this with try - except AttributeError, but this is not my preferred way either.

try:
    value1 = dictionary2.get("optional").get("value1")
except AttributeError:
    value1 = " "

I also don't like checking if optional field exists, this produces garbage code lines like

optional = dictionary2.get("optional")
if optional:
    value1 = optional.get("value1")
else:
    value1 = " "

which seems very non-Pythonic...

I was thinking maybe my approach of just chaining .get()s is wrong in the first place?

Answer 1

In your code here:

try:
    value1 = dictionary2.get("optional").get("value1")
except AttributeError:
    value1 = " "

You can use brackets and except KeyError:

try:
    value1 = dictionary2["optional"]["value1"]
except KeyError:
    value1 = " "

If this is too verbose for the caller, add a helper:

def get_or_default(d, *keys, default=None):
    try:
        for k in keys:
            d = d[k]
    except (KeyError, IndexError):
        return default
    return d

if __name__ == "__main__":
    d = {"a": {"b": {"c": [41, 42]}}}
    print(get_or_default(d, "a", "b", "c", 1)) # => 42
    print(get_or_default(d, "a", "b", "d", default=43)) # => 43

You could also subclass dict and use tuple bracket indexing, like NumPy and Pandas:

class DeepDict(dict):
    def __init__(self, d, default=None):
        self.d = d
        self.default = default

    def __getitem__(self, keys):
        d = self.d
        try:
            for k in keys:
                d = d[k]
        except (KeyError, IndexError):
            return self.default
        return d

    def __setitem__(self, keys, x):
        d = self.d
        for k in keys[:-1]:
            d = d[k]
        d[keys[-1]] = x

if __name__ == "__main__":
    dd = DeepDict({"a": {"b": {"c": [42, 43]}}}, default="foo")
    print(dd["a", "b", "c", 1]) # => 43
    print(dd["a", "b", "c", 11]) # => "foo"
    dd["a", "b", "c", 1] = "banana"
    print(dd["a", "b", "c", 1]) # => "banana"

But there might be an engineering cost to this if it's confusing for other developers, and you'd want to flesh out the other expected methods as described in How to "perfectly" override a dict? (consider this a proof-of-concept sketch). It's best not to be too clever.