How to timeout code being profiled with cProfiler without modifying user code?

Question

Often during my work I write code to read lines from a file and I process those lines one at a time.

Sometimes the line processing is complicated and the file is long, for example, today it takes roughly a minute for processing 200 lines and the total lines in the file are 175k.

I want to figure out which part of my code is taking a long time and for that I decided to use the cProfiler in Python.

The problem is that I can't actually run the whole code because that would take too long, and if I interrupt the process midway through an exit signal then I cProfiler also dies without producing a report and modifying code with logic to die after a certain reading only top K lines is annoying (because I tend to this kind of thing a lot for different types of data in my job.) I want to avoid adding options only for the sake of profiling if possible.

What would be the cleanest way to tell cProfiler to run for 3 minutes, profile what happens, stop and then report its findings?

Answer 1

Step 1: run your script myscript.py under the profiler for 3 minutes, outputting the profiling information to the file prof. On Linux and similar, you can do this with

timeout -s INT 3m python -m cProfile -o prof myscript.py

(Note: if you omit -s INT, SIGTERM is used instead of SIGINT, which seems to work of Python 2 but not on Python 3.) Alternatively, on any system, you should be able to do

python -m cProfile -o prof myscript.py

then press Ctrl-C at the end of 3 minutes.

Step 2: get some statistics from the prof file with something like

python -c "import pstats; pstats.Stats('prof').sort_stats('time').print_stats(20)"