GC Doesn't Delete Circular References in WeakKeyDictionaries?

Question

I have a situation in which I'd like to maintain a mapping from one object to another for as long as the first object exists. My first thought was to use a WeakKeyDictionary.

import weakref
import gc

class M:
    pass

w = weakref.WeakKeyDictionary()
m = M()
w[m] = some_other_object
del m
gc.collect()
print w.keys()

This works fine in most cases. However, in the case where some_other_object is (or has a reference to) m, the M instance will not be garbage collected. (To see an example, replace some_other_object with m)

Of course, if I stored the mapping on the object itself, it would be garbage collected when I deleted it:

import weakref
import gc

class M:
    pass

m = M()
m.circular_reference = m
r = weakref.ref(m)
del m
gc.collect()
print r

Can I achieve the results of the second example using the weakref module (i.e. without mutating m)?

In other words, can I use the weakref module to map an object to itself (or another object that has a strong reference to it) and only keep the object in memory as long as there are other references to it?

Answer 1

You don't really have circular references in these examples. Your circle goes like:

WeakKeyDict -> value -> key -> ...

So the dict keeps the value alive, which in turn keeps the key alive, and through a separate mechanism that does not involve a strong reference, the key instructs the dict to keep the value alive. Since there is no proper reference circle here, the GC never detects any issue. (However, your second example does contain a circular reference, which is why it behaves like the first)

I suspect the only solution to your problem is to ensure that the values of the dict never have strong references (direct or indirect) to any of the keys in the dict. This is similar to what srgerg proposed, but rather than making weak references to all values, you really want references to the keys to be weak. For example:

import weakref
import gc

class M:
    pass

w = weakref.WeakKeyDictionary()
key = M()
val = M()
val.keyRef = weakref.ref(val)
w[key] = val
print len(w)   ## prints 1
del key
gc.collect()
print len(w)   ## prints 0

This way, there are always strong references to the values, but you are carefully controlling references to keys to determine what gets removed from dict. Depending on the complexity of your program, this could be time consuming to implement, since you need to manually track down all references to keys.

But perhaps we can propose a simpler solution if you tell us more about the specific problem..

Answer 2

Dealing with circular references like this is likely to do one's head in, but, undaunted, I shall try to answer your question anyway.

A python dictionary stores references to both its keys and it values. A WeakKeyDictionary stores weak references to its keys, but strong references to its values. So in your first example, if you set w[m] = m the dictionary w has a weak reference to m as a key and a strong reference to m as a value.

Python's weakref module also has a WeakValueDictionary which has strong references to its keys and weak references to its values, but this won't solve your problem.

What you really want is a dictionary that has weak references to both its keys and values. The following code explicitly makes each value in the dictionary a weak reference:

import weakref
import gc

class M():
  pass

class Other():
  pass

m = M()
w = weakref.WeakKeyDictionary()
w[m] = weakref.ref(m)
print len(w.keys()) # prints 1
del m
print len(w.keys()) # prints 0

m = M()
o = Other()
o.strong_ref = m
w[m] = weakref.ref(o)
print len(w.keys()) # prints 1
del m
print len(w.keys()) # prints 1
del o
print len(w.keys()) # prints 0

You could make the value a weak reference automatically by sub-classing WeakKeyDictionary. Note, however, that this does not behave like a real WeakValueDictionary because it does not have a callback to notify the dictionary when a value has been deleted.

class MyWeakKeyValueDictionary(weakref.WeakKeyDictionary):
    def __init__(self):
        weakref.WeakKeyDictionary.__init__(self)

    def __setitem__(self, key, value):
        weakref.WeakKeyDictionary.__setitem__(self, key, weakref.ref(value))

w2 = MyWeakKeyValueDictionary()
m = M()
w2[m] = m
print len(w2.keys()) # prints 1
del m
print len(w2.keys()) # prints 0

m = M()
o = Other()
o.strong_ref = m
w2[m] = o
print len(w2.keys()) # prints 1
del m
print len(w2.keys()) # prints 1
del o
print len(w2.keys()) # prints 0

In the end, however, I fear trying to deal with circular references like this may cause more problems than it's worth.