In-place dictionary inversion in Python

Question

I need to invert a dictionary of lists, I don't know how to explain it in English exactly, so here is some code that does what I want. It just takes too much memory.

def invert(oldDict):
    invertedDict = {}
    for key,valuelist in oldDict.iteritems():
        for value in valuelist:
            try:
                entry = invertedDict[value]
                if key not in entry:
                    entry.append(key)
            except KeyError:
                invertedDict[value] = [key]
    return invertedDict

The original is a dict of lists, and the result is a dict of lists. This "inverts" it.

test = {}
test[1] = [1999,2000,2001]
test[2] = [440,441]
test[3] = [440,2000]

print invert(test)

This gives:

{2000: [1, 3], 2001: [1], 440: [2, 3], 441: [2], 1999: [1]}

I need to know if this can be done in-place, because my current strategy is exceeding the amount of physical memory on my machine with the dictionary I am working with. Can you think of a way to do it with generators?

Answer 1

This doesn't do it in place, but consumes oldDict by using popitem()

from collections import defaultdict
def invert(oldDict):
    invertedDict = defaultdict(list)
    while oldDict:
        key, valuelist = oldDict.popitem()
        for value in valuelist:
            invertedDict[value].append(key)
    return invertedDict

I have a feeling that dict's are never resized unless the size increases, so you may need to add+remove a dummy item periodically. See Shrinkage rate

from collections import defaultdict
def invert(oldDict):
    invertedDict = defaultdict(list)
    i=0
    while oldDict:
        key, valuelist = oldDict.popitem()
        for value in valuelist:
            invertedDict[value].append(key)
        i+=1
        if i%1000==0: # allow the dict to release memory from time to time
            oldDict[None]=None
            del oldDict[None]
    return invertedDict