log a report from memory_profiler

Question

I am profiling my code using memory_profiler

from memory_profiler import profile

@profile
def whatever():
    ....
    ....

So, as many of you may know I am getting an output on the screen similar like this:

Line #    Mem usage  Increment   Line Contents
==============================================
     3                           @profile
     4      5.97 MB    0.00 MB   def my_func():
     5     13.61 MB    7.64 MB       a = [1] * (10 ** 6)
     6    166.20 MB  152.59 MB       b = [2] * (2 * 10 ** 7)
     7     13.61 MB -152.59 MB       del b
     8     13.61 MB    0.00 MB       return a

My question is:

Since the @profile process takes a lot of time, I was wondering if I could somehow log/store this output, and leave the script running, maybe during the night.

My idea is to use the decorator @profile in many def functions, and store all the results somehow in one single TXT or in many different TXT files, this is not importatnt, the important is if that would be possible.

Answer 1

_{^{As from the comments:}}

If you just run

run_my_thing > output.txt

in the shell you can store stdout in a file.

This will circumvent memory_profiler altogether. Obviously it's not ideal to just redirect stdout, but if it's for human analysis it shouldn't be a massive problem.

Answer 2

I haven't tried it, but seems pretty simple - From the docs:

REPORTING

The output can be redirected to a log file by passing IO stream as parameter to the decorator like @profile(stream=fp) .

>>> fp=open('memory_profiler.log','w+')
>>> @profile(stream=fp)
>>> def my_func():
    ...     a = [1] * (10 ** 6)
    ...     b = [2] * (2 * 10 ** 7)
    ...     del b
    ...     return a

---

For many txt/log files i.e. keeping result for various functions/code blocks separately - pass a different file object while decorating a function:

fp=open('memory_profiler.log','w+')
@profile(stream=fp)
def func1():
    # statements

fp2=open('memory_profiler2.log', 'w+')
@profile(stream=fp2)
def func2():
    # statements
.....

Downside: Lot of open connections.

---

Elegant way to log to multiple files is to make use of RotatingFileHandler:

Sometime it would be very convenient to use logger module specially when we need to use RotatingFileHandler. The output can be redirected to logger module by simply making use of LogFile of memory profiler module

.

from memory_profiler import LogFile
import sys

sys.stdout = LogFile('memory_profile_log')