Deep version of sys.getsizeof [duplicate]

Question

I want to calculate the memory used by an object. sys.getsizeof is great, but is shallow (for example, called on a list, it would not include the memory taken by the list's elements).

I'd like to write a generic "deep" version of sys.getsizeof. I understand there is some ambiguity in the definition of "deep"; I'm perfectly happy with the definition followed by copy.deepcopy.

Here's my first attempt:

def get_deep_sizeof(x, level=0, processed=None):
    if processed is None:
        # we're here only if this function is called by client code, not recursively
        processed = set()
    processed.add(id(x))
    mem = sys.getsizeof(x)
    if isinstance(x, collections.Iterable) and not isinstance(x, str):
        for xx in x:
            if id(xx) in processed:
                continue
            mem += get_deep_sizeof(xx, level+1, processed)
            if isinstance(x, dict):
                mem += get_deep_sizeof(x[xx], level+1, processed)
    return mem

It suffers from two known problems, and an unknown number of unknown problems:

• I don't know how to traverse a generic container in a way that captures all the linked objects. Therefore, I iterated using in, and hard coded the case of dictionary (to include values, and not just the keys). Obviously, this will not work for other classes like dictionary.
• I had to hard code the exclusion of str (which is an iterable, and yet does not have links to any other objects). Again, this will break if there are more objects like that.

I suspect that using in is not a good idea, but I'm not sure what else to do.

Answer 1

I think that Pympler has already beaten you to the punch on this one.

From their documentation:

>>> from pympler.asizeof import asizeof
>>> obj = [1, 2, (3, 4), 'text']
>>> asizeof(obj)
176

The source code can be found here.