Xcode tests pass in isolation, fail when run with other tests

Question

I've written some asynchronous unit tests with XCTest expectations to test a networking class I wrote. Most of my tests work every time.

There are a few tests that fail when I run the whole suite, but pass on their own.

Other tests fail, but making requests with the same URLs return appropriate data when pasted into a browser.

My network code is encapsulated in NSOperation objects which are run on an NSOperationQueue. (My operation queue is the default kind - I haven't explicitly set the underlying GCD queue to be serial or concurrent.)

What can I look at to fix these tests? After reading this post on objc.io, I'm assuming they are suffering from some sort of isolation problem.

Answer 1

You're on the right path. The solution suggested by objc.io article is probably The Right Way to do it but does require some refactoring. If you want to make the tests pasts as a first step before you go on a code change binge, here's how you might do it.

Generally you can use the XCTestExpectations to do almost all of your async testing. A standard pattern might go like this:

XCTestExpectation *doThingPromise = [self expetationWithDescription:@"Bazingo"];
[SomeService doThingOnSucceed:^{
  [doThingPromise fulfill];
} onFail:^ {
}];
[self waitForExpectationsWithTimeout:1.0 handler:^(NSError *error) {
    expect(error).to.beNil();
}]

This works fine if [SomeService doThingOnSucceed:onFail:] fires off an async request and then resolves directly. But what if it did more exotic things like:

+ (void)doThingOnSucceed:onFail: {
  [Thing do it];
  [self.context performBlock:^{
    // Uh oh Farfalle-Os
    success();
  }];
}  

The perform block would get set up but you wouldn't be waiting for it to finish because you're not actually waiting on the inner block, just the outer one. The key is that XCTestWaits actually lets the test finish and then just checks that the promise was fulfilled within some time period but in the mean time it will start running other tests. That success() could appear any number of places and produces any number of weird behaviors.

The isolation behavior (vs no isolation) comes from the fact that if you run only this test everything might be fine due to luck but if you run multiple tests that CoreData block might just be stuck hanging around until the next test that is async, which will then "unblock" its execution and it'll start executing at some random future time for some random future test.

The short-term explicit hack around is to pause your test until things finish. Here's an example:

dispatch_semaphore_t semaphore = dispatch_semaphore_create(0);
[SomeService doThingOnComplete:^{
  dispatch_semaphore_signal(semaphore);
}];
while (dispatch_semaphore_wait(semaphore, DISPATCH_TIME_NOW)) {
    [[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode beforeDate:[NSDate dateWithTimeIntervalSinceNow:10]];
}

This explicitly prevents a test from completing until everything finishes, which means no other tests can run until this test finishes.

If there are a lot of these cases in your tests/code, I would recommend the objc.io solution of creating a dispatch group that you can wait on after every test.

Answer 2

After fighting NSOperationQueue and a seemingly incorrect return from waitUntilAllOperationsAreFinished for a couple of days I have hit upon a simpler option: partition your tests into multiple test targets. This gives your tests their own 'app' environment and, more importantly in this case, ensures that Xcode/XCUnit will run them sequentially so that they cannot interfere with each other - unless they do things like leaving a database dirty (which probably should be a failure anyway).

The quickest way to do this is to duplicate your test target, delete the failing tests from the original and delete everything except your failing tests from the new target. Note that if you have multiple tests interfering with each other you many need multiple targets to achieve enough isolation.

You can check that the tests are executed by inspecting the test target scheme. In the scheme you should see both (all) of you test targets and underneath them your individual tests.