python RuntimeError: dictionary changed size during iteration

Question

I have obj like this

{hello: 'world', "foo.0.bar": v1, "foo.0.name": v2, "foo.1.bar": v3}

It should be expand to

{ hello: 'world', foo: [{'bar': v1, 'name': v2}, {bar: v3}]}

I wrote code below, splite by '.', remove old key, append new key if contains '.', but it said RuntimeError: dictionary changed size during iteration

def expand(obj):
    for k in obj.keys():
        expandField(obj, k, v)

def expandField(obj, f, v):
    parts = f.split('.')
    if(len(parts) == 1):
        return
    del obj[f]
    for i in xrange(0, len(parts) - 1):
        f = parts[i]
        currobj = obj.get(f)
        if (currobj == None):
            nextf = parts[i + 1]
            currobj = obj[f] = re.match(r'\d+', nextf) and [] or {}
        obj = currobj
    obj[len(parts) - 1] = v

for k, v in obj.iteritems():

RuntimeError: dictionary changed size during iteration

Answer 1

Like the message says: you changed the number of entries in obj inside of expandField() while in the middle of looping over this entries in expand.

You might try instead creating a new dictionary of the form you wish, or somehow recording the changes you want to make, and then making them AFTER the loop is done.

Answer 2

You might want to copy your keys in a list and iterate over your dict using the latter, eg:

def expand(obj):
    keys = list(obj.keys())  # freeze keys iterator into a list
    for k in keys:
        expandField(obj, k, v)

I let you analyse if the resulting behavior suits your expected results.

Edited as per comments, thank you !

Answer 3

I had a similar issue with wanting to change the dictionary's structure (remove/add) dicts within other dicts.

For my situation I created a deepcopy of the dict. With a deepcopy of my dict, I was able to iterate through and remove keys as needed.Deepcopy - PythonDoc

A deep copy constructs a new compound object and then, recursively, inserts copies into it of the objects found in the original.

Hope this helps!

Answer 4

For those experiencing

RuntimeError: dictionary changed size during iteration

also make sure you're not iterating through a defaultdict when trying to access a non-existent key! I caught myself doing that inside the for loop, which caused the defaultdict to create a default value for this key, causing the aforementioned error.

The solution is to convert your defaultdict to dict before looping through it, i.e.

d = defaultdict(int)
d_new = dict(d)

or make sure you're not adding/removing any keys while iterating through it.