How to automate performance tests and integrate with CI?

Question

I am thinking of automating performance tests, in the same way than what we currently have with unit tests.

I know how to run performance tests with tools like jMeter or by writing my own code to trigger specific parts of the application. I know how to use time, jvisualvm, nmon or others to gather information about resources being used.

I would like to go further and write a performance test, that would fail if it crosses certain lines (execution time, memory or CPU consumed...). I would then have my CI server (Jenkins) run the tests on a regular basis to ensure the performance remains good.

This is complicated because performance depends on the hardware, and in the current way I do it, it requires human interpretation of results to decide whether this is satisfying or not.

Do you know any tools or frameworks (if possible Java based) that help automate performance tests in that way? If not, do you have some good practice to advice?

Thanks.

Answer 1

If your testing is subjective, then to automate it, you need to take some of the subjectivity "out". By that I mean, set some thresholds that you deem acceptable and not-acceptable. See if there is a way to throw a flag or something that Jenkins can pick up on. If you have these thresholds, you stand a better chance of getting the automation you desire.

Answer 2

In the past, I have used JUnit to do some performance testing. However, it did not need human interpretation - the algorithm either took way too long or it was quick enough. In a way, it was a pass/fail test, based on a time threshold.

If you need subjective performance testing done automatically, I am afraid it will be difficult to build.

Answer 3

Jenkins has a "Performance Plugin" That captures results from JMeter and JUnit. Look for it in the "available plugins" under "Plugins" under "Manage Jenkins"

Answer 4

To pass or fail test automatically, you must be able to define pass/fail criteria in terms of numbers of booleans. This can be average response time, or slightly more advanced statistical analysis, e.g. trend function or standard deviation.

I don't know any tools which do this kind of things for both client- and server-side.

There is a limited number of tools which can do that for client-side results.

With Jenkins Performance Plugin you should be able to pass/fail a build, using configured error threshold. This is very basic automatic verification of test results and a way to pass or fail the build.

When I came across similar challenge, I evaluated Performance Plugin and wasn't fully satisfied with functionalities it provides. This way I started working on Java-based Lightning project. It gives you the ability to analyse JMeter results and pass of rail the build automatically, based on e.g. average response times for particular transaction type, or standard deviation.