Python line-by-line memory profiler?

Question

I'm looking to generate, from a large Python codebase, a summary of heap usage or memory allocations over the course of a function's run.

I'm familiar with heapy, and it's served me well for taking "snapshots" of the heap at particular points in my code, but I've found it difficult to generate a "memory-over-time" summary with it. I've also played with line_profiler, but that works with run time, not memory.

My fallback right now is Valgrind with massif, but that lacks a lot of the contextual Python information that both Heapy and line_profiler give. Is there some sort of combination of the latter two that give a sense of memory usage or heap growth over the execution span of a Python program?

Answer 1

I would use sys.settrace at program startup to register a custom tracer function. The custom_trace_function will be called for each line of code. Then you can use that function to store information gathered by heapy or meliae in a file for later processing.

Here is a very simple example which logs the output of hpy.heap() each second to a plain text file:

import sys import time import atexit from guppy import hpy  _last_log_time = time.time() _logfile = open('logfile.txt', 'w')  def heapy_profile(frame, event, arg):     currtime = time.time()     if currtime - _last_log_time < 1:         return     _last_log_time = currtime     code = frame.f_code     filename = code.co_filename     lineno = code.co_firstlineno     idset = hpy().heap()     logfile.write('%s %s:%s\n%s\n\n' % (currtime, filename, lineno, idset))     logfile.flush()  atexit.register(_logfile.close) sys.settrace(heapy_profile) 

Answer 2

You might be interested by memory_profiler.