numpy.ndarray objects not garbage collected

Question

While trying to fine-tune some memory leaks in the Python bindings for some C/C++ functions I cam across some strange behavior pertaining to the garbage collection of Numpy arrays.

I have created a couple of simplified cases in order to better explain the behavior. The code was run using the memory_profiler, the output from which follows immediately after. It appears that Python's garbage collection is not working as expected when it comes to NumPy arrays:

# File deallocate_ndarray.py
@profile
def ndarray_deletion():
    import numpy as np
    from gc import collect
    buf = 'abcdefghijklmnopqrstuvwxyz' * 10000
    arr = np.frombuffer(buf)
    del arr
    del buf
    collect()
    y = [i**2 for i in xrange(10000)]
    del y
    collect()

if __name__=='__main__':
    ndarray_deletion()

With the following command I invoked the memory_profiler:

python -m memory_profiler deallocate_ndarray.py

This is what I got:

Filename: deallocate_ndarray.py
Line #    Mem usage    Increment   Line Contents
================================================
 5   10.379 MiB    0.000 MiB   @profile
 6                             def ndarray_deletion():
 7   17.746 MiB    7.367 MiB       import numpy as np
 8   17.746 MiB    0.000 MiB       from gc import collect
 9   17.996 MiB    0.250 MiB       buf = 'abcdefghijklmnopqrstuvwxyz' * 10000
10   18.004 MiB    0.008 MiB       arr = np.frombuffer(buf)
11   18.004 MiB    0.000 MiB       del arr
12   18.004 MiB    0.000 MiB       del buf
13   18.004 MiB    0.000 MiB       collect()
14   18.359 MiB    0.355 MiB       y = [i**2 for i in xrange(10000)]
15   18.359 MiB    0.000 MiB       del y
16   18.359 MiB    0.000 MiB       collect()

I don't understand why even the forced calls to collect don't reduce the memory usage of the program by freeing up some memory. Moreover, even if Numpy arrays don't behave normally due to the underlying C constructs, why doesn't the list (which is pure Python) get garbage collected?

I know that del does not directly call the underlying __del__ method, but you will note that all del statements in the code actually end up reducing the reference count of the corresponding objects to zero (thereby making them eligible for garbage collection AFAIK). Typically, I would expect to see a negative entry in the increment column when an object undergoes garbage collection. Can anyone shed some light on what is going on here?

NOTE: This test was run on OS X 10.10.4, Python 2.7.10 (conda), Numpy 1.9.2 (conda), Memory Profiler 0.33 (conda-binstar), psutil 2.2.1 (conda).

Answer 1

In order to see the memory garbage collected, I had to increase the size of buf several orders of magnitude. Maybe the size is too small for memory_profiler to detect the change (it queries the OS, so measurements are not very precise) or maybe its too small for the Python garbage collector to care, I don't know.

For example, replacing 10000 by 100000000 in the factor buf yields

Line #    Mem usage    Increment   Line Contents
================================================
21   10.289 MiB    0.000 MiB   @profile
22                             def ndarray_deletion():
23   17.309 MiB    7.020 MiB       import numpy as np
24   17.309 MiB    0.000 MiB       from gc import collect
25 2496.863 MiB 2479.555 MiB       buf = 'abcdefghijklmnopqrstuvwxyz' * 100000000
26 2496.867 MiB    0.004 MiB       arr = np.frombuffer(buf)
27 2496.867 MiB    0.000 MiB       del arr
28   17.312 MiB -2479.555 MiB       del buf
29   17.312 MiB    0.000 MiB       collect()
30   17.719 MiB    0.406 MiB       y = [i**2 for i in xrange(10000)]
31   17.719 MiB    0.000 MiB       del y
32   17.719 MiB    0.000 MiB       collect()