Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to make a performance test fail if it's too slow?

I'd like my test to fail if it runs slower than 0.5 seconds but the average time is merely printed in the console and I cannot find a way to access it. Is there a way to access this data?

Code

//Measures the time it takes to parse the participant codes from the first 100 events in our test data.
func testParticipantCodeParsingPerformance()
{
    var increment = 0
    self.measureBlock
    {
        increment = 0
        while increment < 100
        {
            Parser.parseParticipantCode(self.fields[increment], hostCodes: MasterCalendarArray.getHostCodeArray()[increment])
            increment++
        }
    }
    print("Events measured: \(increment)")
}

Test Data

[Tests.ParserTest testParticipantCodeParsingPerformance]' measured [Time, seconds] average: 0.203, relative standard deviation: 19.951%, values: [0.186405, 0.182292, 0.179966, 0.177797, 0.175820, 0.205763, 0.315636, 0.223014, 0.200362, 0.178165]

like image 731
Declan McKenna Avatar asked Jul 14 '16 13:07

Declan McKenna


People also ask

How do you handle a slow test?

One solution to slow tests is to trade integration for speed. If the system under test calls for a database query, replace the query with a mock that returns a prefab dataset instead.

How long should performance test run?

Ideally, you'd run all the tests in your test suite for at least 10 minutes, if not longer, and that's too long to run for every commit. If you're like us, by the time you've run a load test for 30 or 60 minutes, you've mentally moved on and have switched contexts. Your head's no longer in the code.

What are the common mistakes done in performance testing?

The most common performance testing mistake is to use improper think time and pacing delays. Some either forget to add them or use unrealistic user think time. Many people hit their application with hundreds or thousands of requests per second without any think time and then wonder why the response time is slow.

What is a slow test?

Slow tests affect development. Engineering teams lose momentum and become frustrated because they can't meet their goals. A slow test suite puts the brakes on CI/CD, making release and deployment more difficult. This often means that organizations can't ship out products on time, and risk losing their competitive edge.


1 Answers

You need to set a baseline for your performance test. Head to the Report Navigator:

Report Navigator

and select your recent test run. You'll see a list of all your tests, but the performance ones will have times associated with them. Click the time to bring up the Performance Result popover:

Performance Result

The "Baseline" value is what you're looking for--set it to 0.5s and that will inform Xcode that this test should complete in half a second. If your test is more than 10% slower than the baseline, it'll fail!

like image 125
andyvn22 Avatar answered Sep 28 '22 15:09

andyvn22