Python WeakValueDictionary retains values after gc.collect() in IPython

Question

I'm trying to understand Python weakref module and its use cases, so I have the following setup:

import gc, weakref

class obj(object):
    def __init__(self, val=None):
        self._s = "Sample" if not val else " ".join(["Sample:", str(val)])
    def sample(self):
        return self._s

ol = [obj(x) for x in range(1,4)]
o1 = obj(1)
o2 = obj(2)
o3 = obj(3)

wdict1 = weakref.WeakValueDictionary({k:ol[k-1] for k in range(1,4)})

wdict2 = weakref.WeakValueDictionary()
wdict2[1] = o1
wdict2[2] = o2
wdict2[3] = o3

After running my tests I can clearly see, that WeakValueDictionary somehow has retained all the values, even though I have explicitly called gc.collect(). To my understanding and as explained in the following answer, calling gc.collect() should remove all weakly referenced values.

In [2]: wdict1.items()
Out[2]: 
[(1, <__main__.obj at 0x7fea09c0be90>),
 (2, <__main__.obj at 0x7fea09c0bf10>),
 (3, <__main__.obj at 0x7fea09c0bf50>)]

In [3]: wdict2.items()
Out[3]: 
[(1, <__main__.obj at 0x7fea09c51790>),
 (2, <__main__.obj at 0x7fea09c0bed0>),
 (3, <__main__.obj at 0x7fea09c0bf90>)]

In [4]: del ol[0]

In [5]: del o1

In [6]: gc.collect()
Out[6]: 64

In [7]: wdict1.items()
Out[7]: 
[(1, <__main__.obj at 0x7fea09c0be90>),
 (2, <__main__.obj at 0x7fea09c0bf10>),
 (3, <__main__.obj at 0x7fea09c0bf50>)]

In [8]: wdict2.items()
Out[8]: 
[(1, <__main__.obj at 0x7fea09c51790>),
 (2, <__main__.obj at 0x7fea09c0bed0>),
 (3, <__main__.obj at 0x7fea09c0bf90>)]

In [9]: del ol[0]

In [10]: del o2

In [11]: gc.collect()
Out[11]: 0

In [12]: wdict1.items()
Out[12]: 
[(1, <__main__.obj at 0x7fea09c0be90>),
 (2, <__main__.obj at 0x7fea09c0bf10>),
 (3, <__main__.obj at 0x7fea09c0bf50>)]

In [13]: wdict2.items()
Out[13]: 
[(1, <__main__.obj at 0x7fea09c51790>),
 (2, <__main__.obj at 0x7fea09c0bed0>),
 (3, <__main__.obj at 0x7fea09c0bf90>)]

In [14]: weakref.getweakrefs(ol[0])
Out[14]: [<weakref at 0x7fea0ab05470; to 'obj' at 0x7fea09c0bf50>]

In [15]: weakref.getweakrefs(o3)
Out[15]: [<weakref at 0x7fea09c060b0; to 'obj' at 0x7fea09c0bf90>]

In [16]: wdict1[1].sample()
Out[16]: 'Sample: 1'

In [17]: wdict2[2].sample()
Out[17]: 'Sample: 2'

What's wrong with my code, why all the weakly referenced values were kept?

Answer 1

The issue is that IPython keeps references to your objects using the following -

1. _ - result of last statement.

2. __ - result of second last statement.

3. ___ - result of third last statement.

4. In[num] - string of the statement that run for prompt number num

5. Out[num] - result/output of the prompt number num statement.

From documentation -

Input and output history are kept in variables called In and Out, keyed by the prompt numbers, e.g. In[4]. The last three objects in output history are also kept in variables named _, __ and ___.

You can actually try running Out[2] and see that it would be the reference to the result of wdict1.items() (if in 2 prompt number you ran that statement as you have given in your example).

IPython is most probably keeping alot of such references to your object, hence when you delete one of the names like ol[0] or o1 , and then do gc.collect . It does not actually collect the object , since there are still references to the object (in _ or __ or ___ or Out[num] ).

Two solutions I can think of -

1. Use %xdel magic command to remove the reference, like %xdel ol[0] , instead of del ol[0] . This would cause IPython to clear out all references it keeps as well. Based on documentation -

Delete a variable, trying to clear it from anywhere that IPython’s machinery has references to it.

2. You can try running this tests as a script ,instead of interactively, in that case these extra references would not be created, and you should see the correct behavior.