Profiling python C extensions

Question

I have developed a python C-extension that receives data from python and compute some cpu intensive calculations. It's possible to profile the C-extension?

The problem here is that writing a sample test in C to be profiled would be challenging because the code rely on particular inputs and data structures (generated by python control code).

Do you have any suggestions?

Answer 1

After the comment by pygabriel I decided to upload a package to pypi that implements a profiler for python extensions using the cpu-profiler from google-perftools: http://pypi.python.org/pypi/yep

Answer 2

I've found my way using google-perftools. The trick was to wrap the functions StartProfiler and StopProfiler in python (throught cython in my case).

To profile the C extension is sufficient to wrap the python code inside the StartProfiler and StopProfiler calls.

from google_perftools_wrapped import StartProfiler, StopProfiler import c_extension # extension to profile c_extension.so  StartProfiler("output.prof") ... calling the interesting functions from the C extension module ... StopProfiler() 

Then to analyze for example you can export in callgrind format and see the result in kcachegrind:

pprof --callgrind c_extension.so output.prof > output.callgrind  kcachegrind output.callgrind