How to profile my code?

Question

I want to know how to profile my code.

I have gone through the docs, but as there were no examples given I could not get anything from it.

I have a large code and it is taking so much time, hence I want to profile and increase its speed. I havent written my code in method, there are few in between but not completely. I don't have any main in my code. I want to know how to use profiling. I'm looking for some example or sample code of about how to profile.

I tried psyco, i.e just addded two lines at the top of my code:

import psyco psyco.full() 

Is this right? It did not show any improvement. Any other way of speeding up, please suggest.

Answer 1

The standard answer to this question is to use cProfile.

You'll find though that without having your code separated out into methods that cProfile won't give you particularly rich information.

Instead, you might like to try what another poster here calls Monte Carlo Profiling. To quote from another answer:

If you're in a hurry and you can manually interrupt your program under the debugger while it's being subjectively slow, there's a simple way to find performance problems.

Just halt it several times, and each time look at the call stack. If there is some code that is wasting some percentage of the time, 20% or 50% or whatever, that is the probability that you will catch it in the act on each sample. So that is roughly the percentage of samples on which you will see it. There is no educated guesswork required. If you do have a guess as to what the problem is, this will prove or disprove it.

You may have multiple performance problems of different sizes. If you clean out any one of them, the remaining ones will take a larger percentage, and be easier to spot, on subsequent passes.

Caveat: programmers tend to be skeptical of this technique unless they've used it themselves. They will say that profilers give you this information, but that is only true if they sample the entire call stack. Call graphs don't give you the same information, because 1) they don't summarize at the instruction level, and 2) they give confusing summaries in the presence of recursion. They will also say it only works on toy programs, when actually it works on any program, and it seems to work better on bigger programs, because they tend to have more problems to find [emphasis added].

It's not orthodox, but I've used it very successfully in a project where profiling using cProfile was not giving me useful output.

The best thing about it is that this is dead easy to do in Python. Simply run your Python script in the interpreter, press [Control-C], note the traceback and repeat a number of times.

Answer 2

Edit:

This answer has been implemented in https://github.com/campos-ddc/cprofile_graph

Profiling with cProfile

Here's a post I wrote some time ago on profiling with cProfile with some graphical aid.

cProfile is one of the most used python profilers out there, and although very powerful, the standard text output is somewhat lackluster. Here I'll show you how to use cProfile on your application in an easier way.

There are two common ways to use cProfile, you can use it as a command in prompt to profile a given module, or you can use it inside your code, to profile specific snippets of code.

Profiling a module

To use cProfile to profile an entire module, simply use the following command in your prompt:

python -m cProfile -o output_filename.pstats path/to/script arg1 arg2 

This will run your module with the given arguments (they are optional) and dump the output in output_filename.pstats.

There are lots of ways to read the data on that output file, but for the purpose of this post, let's not worry about those and focus on getting that graphical visualization.

Profiling from inside

Sometimes you don't want to profile an entire module, just a few lines of it.

To do so, you are gonna have to add some code to your module.

First of all:

import cProfile 

And then, you can replace any segment of code with the following:

cProfile.runctx('Your code here', globals(), locals(), 'output_file') 

For example, here is a test before and after profiling:

import unittest  class Test(unittest.TestCase):      def testSomething(self):         self.DoSomethingIDontCareAbout()          param = 'whatever'         self.RunFunctionIThinkIsSlow(param)          self.AssertSomeStuff() # This is after all, a test 

After:

import unittest import cProfile  class Test(unittest.TestCase):      def testSomething(self):         self.DoSomethingIDontCareAbout()          param = 'whatever'         cProfile.runctx(             'self.RunFunctionIThinkIsSlow(param)',             globals(),             locals(),             'myProfilingFile.pstats'         )          self.AssertSomeStuff() # This is after all, a test 

Converting a pstats file to a graph

To convert your profiling file to a graph, you will need a couple of things:

• gprof2dot: This module will convert your output into a dot file, a standard file format for graph descriptions.
• GraphViz: It turns your dot file into an image.

After you have downloaded gprof2dot and installed GraphViz, run this command in your prompt:

python gprof2dot -f pstats myProfileFile | dot -Tpng -o image_output.png 

You might have to use a complete path for gprof2dot and/or dot, or you could add them to your PATH env variable.

After all of this, you should have an image that looks kinda like this:

• Hotter colors (red, orange, yellow) indicate functions that take up more of the total runtime than colder colors (green, blue)

• On each node, you can see what percentage of the total runtime that function used and how many times it was called.

• Arrows between nodes indicate which function called other functions, and such arrows also have a caption indicating what percentage of the runtime came through there.

Note: percentages won't always add up to 100%, especially on code sections that reference C++ code, which won't be profiled. cProfile also won't be able to determine what's called from inside an "eval" statement, so you might see some jumps in your graph.