Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python: How to estimate / calculate memory footprint of data structures?

What's a good way to estimate the memory footprint of an object?

Conversely, what's a good way to measure the footprint?

For example, say I have a dictionary whose values are lists of integer,float tuples:

d['key'] = [ (1131, 3.11e18), (9813, 2.48e19), (4991, 9.11e18) ]

I have 4G of physical memory and would like to figure out approximately how many rows (key:values) I can store in memory before I spill into swap. This is on linux/ubuntu 8.04 and OS X 10.5.6 .

Also, what's the best way to figure out the actual in-memory footprint of my program? How do I best figure out when it's exhausting physical memory and spilling?

like image 532
Parand Avatar asked Apr 14 '09 22:04

Parand


People also ask

How does Python calculate memory consumption?

You can use it by putting the @profile decorator around any function or method and running python -m memory_profiler myscript. You'll see line-by-line memory usage once your script exits.

How do you get the size of a data structure in Python?

In python, the usage of sys. getsizeof() can be done to find the storage size of a particular object that occupies some space in the memory. This function returns the size of the object in bytes.

How much memory does a class occupy in Python?

When you create a list object, the list object by itself takes 64 bytes of memory, and each item adds 8 bytes of memory to the size of the list because of references to other objects.

How do I find a memory leak in Python?

You can detect memory leaks in Python by monitoring your Python app's performance via an Application Performance Monitoring tool such as Scout APM. Once you detect a memory leak, there are multiple ways to solve it.


2 Answers

Guppy has a nice memory profiler (Heapy):

>>> from guppy import hpy
>>> hp = hpy()
>>> hp.setrelheap() # ignore all existing objects
>>> d = {}
>>> d['key'] = [ (1131, 3.11e18), (9813, 2.48e19), (4991, 9.11e18) ]
>>> hp.heap()
 Partition of a set of 24 objects. Total size = 1464 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      2   8      676  46       676  46 types.FrameType
     1      6  25      220  15       896  61 str
     2      6  25      184  13      1080  74 tuple
 ...

Heapy is a little underdocumented, so you might have to dig through the web page or source code a little, but it's very powerful. There are also some articles which might be relevant.

like image 50
Torsten Marek Avatar answered Sep 17 '22 20:09

Torsten Marek


You can do this with a memory profiler, of which there are a couple I'm aware of:

  1. PySizer - poissibly obsolete, as the homepage now recommends:

  2. Heapy.

This is possibly a duplicate of this question.

like image 22
Brian Avatar answered Sep 18 '22 20:09

Brian