Can I run line_profiler over a pytest test?

Question

I have identified some long running pytest tests with

py.test --durations=10 

I would like to instrument one of those tests now with something like line_profiler or cprofile. I really want to get the profile data from the test itself as the pytest setup or tear down could well be part of what is slow.

However given how line_profiler or cprofile is typically involved it isn't clear to me how to make them work with pytest.

Answer 1

Run pytest like this:

python3 -m cProfile -o profile -m pytest 

You can even pass in optional arguments:

python3 -m cProfile -o profile -m pytest tests/worker/test_tasks.py -s campaigns 

This will create a binary file called profile in your current directory. This can be analyzed with pstats:

import pstats p = pstats.Stats('profile') p.strip_dirs() p.sort_stats('cumtime') p.print_stats(50) 

This will print the 50 lines with the longest cumulative duration.

Answer 2

To get cProfile and line_profiler to work with py.test code, I did two things:

1. Extended the py.test test code with a call to pytest.main(), which made it executable with the python interpreter as the main driver:

   # pytest_test.py: @profile # for line_profiler only def test_example():     x = 3**32     assert x == 1853020188851841  # for profiling with cProfile and line_profiler import pytest pytest.main(__file__) 

   Now you can run this test without py.test as the main driver using other tools:

   $ kernprof.py -l pytest_test.py $ python -m line_profiler pytest_test.py.lprof 

   or

   $ python -m cProfile pytest_test.py 

2. To profile py.test-specific functions such as pytest_funcarg*() with line_profiler I split them in two to avoid confusion between py.test and line_profiler:

   def pytest_funcarg__foo(request):     return foo(request)  @profile def foo(request): ... 

The same method works for memory_profiler.