python: How to trace function execution order in large project

Question

I want to trace function/class executive order in scrapy framework. There are multiple *.py files across the default project, and I want to know which py file and class has been executed in order. It sound silly to put logger line in every class and function. How to visualize this order?

cprofile is mainly used for measuring total time. I could also visualize the execution order inside one module, which is common question, but visualizing multiple modules are difficult.

In terms of trace package, I did not find appropriate examples to work with large project like scrapy or django. Trace usage tutorial is about a single python file.

I want to trace multiple *.py files in multiple modules in a large project, eg scrapy, instead of just one module.

I am aware of debug tools like pdb, but I find it cumbersome to put break point across the whole project. More importantly, it is not easy to summarize the execution order.

Finally I solved by using Hunter, which is better than build-in trace module. Trace module did not offer include_dir attribute.

For those who are curiosity about how to trace all lines of scrapy.

$PYTHONHUNTER='Q(module_startswith=["scrapy", "your_project"])' scrapy list 

In terms of django, tracing execution codes of rest_framework and save to test.log, for example:

$PYTHONHUNTER='Q(module_startswith=["rest_framework", "your_project"]), action=CallPrinter(stream=open("test.log", "w"))' python manage.py runserver --noreload --nothreading

Answer 1

Well the best tool to trace function execution order is definitely viztracer. I would have to say that visualization is a huge factor when it comes to understanding a larger project.

An interactive image like this makes it much easier to understand what's going on in your program, compared to cold terminal ascii.

Also, it's a non-intrusive tool, which means you don't need to write a single line of code. Just install it and run your program with it.

pip install viztracer
viztracer your_script.py

Another important factor here is that viztracer supports multi-thread and multi-process, and can visualize them in separate signals, on the same timeline, which you'll never achieve with terminal display.

Answer 2

trace

The trace module allows you to trace program execution, generate annotated statement coverage listings, print caller/callee relationships and list functions executed during a program run. It can be used in another program or from the command line.

python -m trace --count -C . somefile.py ...

The above will execute somefile.py and generate annotated listings of all Python modules imported during the execution into the current directory.

PDB

The module pdb defines an interactive source code debugger for Python programs. It supports setting (conditional) breakpoints and single stepping at the source line level, inspection of stack frames, source code listing, and evaluation of arbitrary Python code in the context of any stack frame. It also supports post-mortem debugging and can be called under program control.

Most Common Used Command:

w(here)

• Print a stack trace, with the most recent frame at the bottom. An arrow indicates the current frame, which determines the context of most commands.

d(own)

• Move the current frame one level down in the stack trace (to a newer frame).

u(p)

• Move the current frame one level up in the stack trace (to an older frame).

You can also check this question Python debugging tips

Coverage

Coverage.py measures code coverage, typically during test execution. It uses the code analysis tools and tracing hooks provided in the Python standard library to determine which lines are executable, and which have been executed.

Hunter

Hunter is a flexible code tracing toolkit, not for measuring coverage, but for debugging, logging, inspection and other nefarious purposes.

The default action is to just print the code being executed. Example:

import hunter
hunter.trace(module='posixpath')

import os
os.path.join('a', 'b')

Result in terminal: