Automatically Running a Test Case Many Times in Xcode

Question

In Xcode, is there a way for me run a single test case n times automatically?

Reason for doing this is that some of my beta testers are encountering random crashes in my app. I see the crash logs in TestFlight, along with the stack trace, but I can't reproduce the crash.

The crash happens infrequently but when it does, it always happens when users are trying to create a DB record, which then gets uploaded to a server. The problem with the crash logs is that my code does not make an appearance in their stack traces (all UIKit & CoreFoundation stuff - and different each time).

My solution is to run the test for that part of the app 100s of times, with the exception breakpoint set, to try to trigger the bug in my dev environment. But I don't know how to do this automatically.

Answer 1

You can read my answer here.

Basically you need to override invokeTest method

override func invokeTest() {
    for time in 0...15 {
        print("this test is being invoked: \(time) times")
        super.invokeTest()
    }
}

Answer 2

It took 7 years, but as of Xcode 13, support for test repetition is now built-in.

From the Xcode 13 release notes:

Enable test repetition in your test plan, xcodebuild, or by running your test from the test diamond by Control-clicking and selecting Run Repeatedly to bring up the test repetition dialog.

Answer 3

In Xcode as such, no.

You can create an XCTestCase class that hooks into the test-running methods it inherits to return multiple runs, but that tends to be annoying and mostly undocumented.

It's probably easier to instead have a "meta-test" that calls out to your other test method repeatedly:

func testOnce() {}

func testManyTimes() {
    for _ in 0..<1000 { testOnce() }
}

You might need to call out to some per-test setup/teardown methods. You could perhaps work around that by instead making the loop body be something like:

let test = XCTestCase(selector: #selector(testOnce))
test.invokeTest()

This would lean on the XCTest machinery that your standard tests use, but it might gripe about not being wired into an XCTestCaseRun (or not).