Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

cProfile and Python: Finding the specific line number that code spends most time on

I'm using cProfile, pstats and Gprof2dot to profile a rather long python script.

The results tell me that the most time is spent calling a method in an object I've defined. However, what I would really like is to know exactly what line number within that function is eating up the time.

Any idea's how to get this additional information?

(By the way, I'm using Python 2.6 on OSX snow leopard if that helps...)

like image 855
Piyush Jain Avatar asked Sep 30 '09 20:09

Piyush Jain


People also ask

How do you use cProfile in Python?

The syntax is cProfile. run(statement, filename=None, sort=-1) . You can pass python code or a function name that you want to profile as a string to the statement argument. If you want to save the output in a file, it can be passed to the filename argument.

Which tool will analyze the data collected by the Python profiler?

Analysis of the profiler data is done using the Stats class. This class constructor creates an instance of a “statistics object” from a filename (or list of filenames) or from a Profile instance.

How does Python code find bottlenecks?

In order to find out the performance bottlenecks in the production applications, developers need actionable insights. A modern approach is to apply profiling that highlights the slowest code that is the area consuming most of your resources like CPU and memory.

Which Python module can profile Python code in a sense to analyze its performance?

In Python's standard library, we have the timeit module that allows us to do some simple profiling. The output of timeit is to find the best performance among multiple runs (default to be 5).


2 Answers

There is a line profiler in python written by Robert Kern.

like image 92
David Cournapeau Avatar answered Sep 30 '22 12:09

David Cournapeau


cProfile does not track line numbers within a function; it only tracks the line number of where the function was defined.

cProfile attempts to duplicate the behavior of profile (which is pure Python). profile uses pstats to store the data from running, and pstats only stores line numbers for function definitions, not for individual Python statements.

If you need to figure out with finer granularity what is eating all your time, then you need to refactor your big function into several, smaller functions.

like image 36
Mark Rushakoff Avatar answered Sep 30 '22 10:09

Mark Rushakoff