Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python: memory usage statistics per object-types (or source code line)

I am doing some heavy calculations with Python (using OpenCV and Numpy) and in the end, I end up with a lot of memory usage (>1GB) whereby all refs should be gone and I only have the end-result (which should not be more than a few MB).

To debug this, it would be nice if I could get some stats somehow which show me how much object instances there are of what type, ordered by the total amount of memory they take (per object class).

Or even nicer: Not per object class but per source code line where the object was created (whereby I guess this info is not available unless I activate some debugging in Python which would make the calculation too slow, so I am not sure if that would be helpful).

Can I get some stats like this somehow? Or how would I debug this?


Some has missunderstood me: I only need to know how to debug the memory usage. Processing/run- time is perfect.

like image 338
Albert Avatar asked Nov 14 '10 15:11

Albert


People also ask

How much memory does a Python object use?

When you create a list object, the list object by itself takes 64 bytes of memory, and each item adds 8 bytes of memory to the size of the list because of references to other objects.

How does Python determine memory usage?

You can use it by putting the @profile decorator around any function or method and running python -m memory_profiler myscript. You'll see line-by-line memory usage once your script exits.

What is memory profiler in Python explain how the Profiler performs a line-by-line analysis of memory?

Memory Profiler is a pure Python module that uses the psutil module. It monitors the memory consumption of a Python job process. Also, it performs a line-by-line analysis of the memory consumption of the application. The line-by-line memory usage mode works in the same way as the line_profiler.

How much memory is allocated to Python?

Python doesn't limit memory usage on your program. It will allocate as much memory as your program needs until your computer is out of memory.


1 Answers

I think you're searching for a python profiler ;

you have a bunch of them that you can use , like Heapy, profile or cprofile , Pysize ...

example using Heapy :

you have to include this snippet somewhere in your code:

from guppy import hpy
h = hpy()
print h.heap()

and it will give you as output:

Partition of a set of 132527 objects. Total size = 8301532 bytes.
Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
0  35144  27  2140412  26   2140412  26 str
1  38397  29  1309020  16   3449432  42 tuple
2    530   0   739856   9   4189288  50 dict (no owner)

example with cprofile :

you can run it like this:

python -m cProfile script.py

Output:

         5 function calls in 0.000 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 myscript.py:1(<module>)
        1    0.000    0.000    0.000    0.000 {execfile}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    0.000    0.000    0.000    0.000 {range}

You can also use gc module to know why python is not freeing your memory, and to ask him to free memory using gc.collect().

By the way have you looked at numpy, i think it more suitable if you're doing heavy calculation like you said.

like image 183
mouad Avatar answered Sep 20 '22 03:09

mouad