Use time elapsed as assertion in unit tests

Question

I am facing a situation in which I want to ensure that if a method is modified it does not takes more than X ms to run (essentially if that function is slower, it will slow down the search results on our web page, and that has a bad impact on sales).

We have unit tests (in particular the code is done under python and we use py.test), the first idea is to assert that if the function did not execute in less than X ms, then mark the test as failed (or raise a warning).

However, this feels dangerous (not all computers are the same speed, for example), and also I am not quite sure that's the job of a unit test.

Has someone faced a similar situation? For me speed is a feature, and I want to ensure that such feature is not lost in the future as the code evolves.

If unit testing is not the answer, which other alternatives would you recommend?

Thanks

Answer 1

You could look into using the pytest-timeout plugin for a start. It currently only allows specifying seconds (as floats). But it could be extended to work with more adaptable timings such as pystone numbers so there would be a measurement of how fast your computer performs the pystone benchmark which could be cached and then you'd specify how many "pystones" your test(s) is allowed to take.