Python recursive setattr()-like function for working with nested dictionaries

Question

There are a lot of good getattr()-like functions for parsing nested dictionary structures, such as:

• Finding a key recursively in a dictionary
• Suppose I have a python dictionary , many nests
• https://gist.github.com/mittenchops/5664038

I would like to make a parallel setattr(). Essentially, given:

cmd = 'f[0].a'
val = 'whatever'
x = {"a":"stuff"}

I'd like to produce a function such that I can assign:

x['f'][0]['a'] = val

More or less, this would work the same way as:

setattr(x,'f[0].a',val)

to yield:

>>> x
{"a":"stuff","f":[{"a":"whatever"}]}

I'm currently calling it setByDot():

setByDot(x,'f[0].a',val)

One problem with this is that if a key in the middle doesn't exist, you need to check for and make an intermediate key if it doesn't exist---ie, for the above:

>>> x = {"a":"stuff"}
>>> x['f'][0]['a'] = val
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'f'

So, you first have to make:

>>> x['f']=[{}]
>>> x
{'a': 'stuff', 'f': [{}]}
>>> x['f'][0]['a']=val
>>> x
{'a': 'stuff', 'f': [{'a': 'whatever'}]}

Another is that keying for when the next item is a lists will be different than the keying when the next item is a string, ie:

>>> x = {"a":"stuff"}
>>> x['f']=['']
>>> x['f'][0]['a']=val
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'str' object does not support item assignment

...fails because the assignment was for a null string instead of a null dict. The null dict will be the right assignment for every non-list in dict until the very last one---which may be a list, or a value.

A second problem, pointed out in the comments below by @TokenMacGuy, is that when you have to create a list that does not exist, you may have to create an awful lot of blank values. So,

setattr(x,'f[10].a',val)

---may mean the algorithm will have to make an intermediate like:

>>> x['f']=[{},{},{},{},{},{},{},{},{},{},{}]
>>> x['f'][10]['a']=val

to yield

>>> x 
{"a":"stuff","f":[{},{},{},{},{},{},{},{},{},{},{"a":"whatever"}]}

such that this is the setter associated with the getter...

>>> getByDot(x,"f[10].a")
"whatever"

More importantly, the intermediates should /not/ overwrite values that already exist.

Below is the junky idea I have so far---I can identify the lists versus dicts and other data types, and create them where they do not exist. However, I don't see (a) where to put the recursive call, or (b) how to 'build' the deep object as I iterate through the list, and (c) how to distinguish the /probing/ I'm doing as I construct the deep object from the /setting/ I have to do when I reach the end of the stack.

def setByDot(obj,ref,newval):
    ref = ref.replace("[",".[")
    cmd = ref.split('.')
    numkeys = len(cmd)
    count = 0
    for c in cmd:
        count = count+1
        while count < numkeys:
            if c.find("["):
                idstart = c.find("[")
                numend = c.find("]")
                try:
                    deep = obj[int(idstart+1:numend-1)]
                except:
                    obj[int(idstart+1:numend-1)] = []
                    deep = obj[int(idstart+1:numend-1)]
            else:
                try:
                    deep = obj[c]
                except:
                    if obj[c] isinstance(dict):
                        obj[c] = {}
                    else:
                        obj[c] = ''
                    deep = obj[c]
        setByDot(deep,c,newval)

This seems very tricky because you kind of have to look-ahead to check the type of the /next/ object if you're making place-holders, and you have to look-behind to build a path up as you go.

UPDATE

I recently had this question answered, too, which might be relevant or helpful.

Answer 1

I have separated this out into two steps. In the first step, the query string is broken down into a series of instructions. This way the problem is decoupled, we can view the instructions before running them, and there is no need for recursive calls.

def build_instructions(obj, q):
    """
    Breaks down a query string into a series of actionable instructions.

    Each instruction is a (_type, arg) tuple.
    arg -- The key used for the __getitem__ or __setitem__ call on
           the current object.
    _type -- Used to determine the data type for the value of
             obj.__getitem__(arg)

    If a key/index is missing, _type is used to initialize an empty value.
    In this way _type provides the ability to
    """
    arg = []
    _type = None
    instructions = []
    for i, ch in enumerate(q):
        if ch == "[":
            # Begin list query
            if _type is not None:
                arg = "".join(arg)
                if _type == list and arg.isalpha():
                    _type = dict
                instructions.append((_type, arg))
                _type, arg = None, []
            _type = list
        elif ch == ".":
            # Begin dict query
            if _type is not None:
                arg = "".join(arg)
                if _type == list and arg.isalpha():
                    _type = dict
                instructions.append((_type, arg))
                _type, arg = None, []

            _type = dict
        elif ch.isalnum():
            if i == 0:
                # Query begins with alphanum, assume dict access
                _type = type(obj)

            # Fill out args
            arg.append(ch)
        else:
            TypeError("Unrecognized character: {}".format(ch))

    if _type is not None:
        # Finish up last query
        instructions.append((_type, "".join(arg)))

    return instructions

For your example

>>> x = {"a": "stuff"}
>>> print(build_instructions(x, "f[0].a"))
[(<type 'dict'>, 'f'), (<type 'list'>, '0'), (<type 'dict'>, 'a')]

The expected return value is simply the _type (first item) of the next tuple in the instructions. This is very important because it allows us to correctly initialize/reconstruct missing keys.

This means that our first instruction operates on a dict, either sets or gets the key 'f', and is expected to return a list. Similarly, our second instruction operates on a list, either sets or gets the index 0 and is expected to return a dict.

Now let's create our _setattr function. This gets the proper instructions and goes through them, creating key-value pairs as necessary. Finally, it also sets the val we give it.

def _setattr(obj, query, val):
    """
    This is a special setattr function that will take in a string query,
    interpret it, add the appropriate data structure to obj, and set val.

    We only define two actions that are available in our query string:
    .x -- dict.__setitem__(x, ...)
    [x] -- list.__setitem__(x, ...) OR dict.__setitem__(x, ...)
           the calling context determines how this is interpreted.
    """
    instructions = build_instructions(obj, query)
    for i, (_, arg) in enumerate(instructions[:-1]):
        _type = instructions[i + 1][0]
        obj = _set(obj, _type, arg)

    _type, arg = instructions[-1]
    _set(obj, _type, arg, val)

def _set(obj, _type, arg, val=None):
    """
    Helper function for calling obj.__setitem__(arg, val or _type()).
    """
    if val is not None:
        # Time to set our value
        _type = type(val)

    if isinstance(obj, dict):
        if arg not in obj:
            # If key isn't in obj, initialize it with _type()
            # or set it with val
            obj[arg] = (_type() if val is None else val)
        obj = obj[arg]
    elif isinstance(obj, list):
        n = len(obj)
        arg = int(arg)
        if n > arg:
            obj[arg] = (_type() if val is None else val)
        else:
            # Need to amplify our list, initialize empty values with _type()
            obj.extend([_type() for x in range(arg - n + 1)])
        obj = obj[arg]
    return obj

And just because we can, here's a _getattr function.

def _getattr(obj, query):
    """
    Very similar to _setattr. Instead of setting attributes they will be
    returned. As expected, an error will be raised if a __getitem__ call
    fails.
    """
    instructions = build_instructions(obj, query)
    for i, (_, arg) in enumerate(instructions[:-1]):
        _type = instructions[i + 1][0]
        obj = _get(obj, _type, arg)

    _type, arg = instructions[-1]
    return _get(obj, _type, arg)


def _get(obj, _type, arg):
    """
    Helper function for calling obj.__getitem__(arg).
    """
    if isinstance(obj, dict):
        obj = obj[arg]
    elif isinstance(obj, list):
        arg = int(arg)
        obj = obj[arg]
    return obj

In action:

>>> x = {"a": "stuff"}
>>> _setattr(x, "f[0].a", "test")
>>> print x
{'a': 'stuff', 'f': [{'a': 'test'}]}
>>> print _getattr(x, "f[0].a")
"test"

>>> x = ["one", "two"]
>>> _setattr(x, "3[0].a", "test")
>>> print x
['one', 'two', [], [{'a': 'test'}]]
>>> print _getattr(x, "3[0].a")
"test"

Now for some cool stuff. Unlike python, our _setattr function can set unhashable dict keys.

x = []
_setattr(x, "1.4", "asdf")
print x
[{}, {'4': 'asdf'}]  # A list, which isn't hashable

>>> y = {"a": "stuff"}
>>> _setattr(y, "f[1.4]", "test")  # We're indexing f with 1.4, which is a list!
>>> print y
{'a': 'stuff', 'f': [{}, {'4': 'test'}]}
>>> print _getattr(y, "f[1.4]")  # Works for _getattr too
"test"

We aren't really using unhashable dict keys, but it looks like we are in our query language so who cares, right!

Finally, you can run multiple _setattr calls on the same object, just give it a try yourself.