XCTestExpectation: how to avoid calling the fulfill method after the wait context has ended?

Question

I’m using the new asynchronous testing capabilities of Xcode 6. Everything works fine when the asynchronous task ends before the timeout. But if the task takes longer than the timeout, things get more complicated.

Here is how I’m doing my tests:

@interface AsyncTestCase : XCTestCase @end

@implementation AsyncTestCase

// The asynchronous task would obviously be more complex in a real world scenario.
- (void) startAsynchronousTaskWithDuration:(NSTimeInterval)duration completionHandler:(void (^)(id result, NSError *error))completionHandler
{
    dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(duration * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{
        completionHandler([NSObject new], nil);
    });
}

- (void) test1TaskLongerThanTimeout
{
    XCTestExpectation *expectation = [self expectationWithDescription:@"Test 1: task longer than timeout"];
    [self startAsynchronousTaskWithDuration:4 completionHandler:^(id result, NSError *error) {
        XCTAssertNotNil(result);
        XCTAssertNil(error);
        [expectation fulfill];
    }];
    [self waitForExpectationsWithTimeout:2 handler:nil];
}

- (void) test2TaskShorterThanTimeout
{
    XCTestExpectation *expectation = [self expectationWithDescription:@"Test 2: task shorter than timeout"];
    [self startAsynchronousTaskWithDuration:5 completionHandler:^(id result, NSError *error) {
        XCTAssertNotNil(result);
        XCTAssertNil(error);
        [expectation fulfill];
    }];
    [self waitForExpectationsWithTimeout:10 handler:nil];
}

@end

Unfortunately, calling the fulfill method after the timeout has expired crashes the test suite with this error:

API violation - called -[XCTestExpectation fulfill] after the wait context has ended.

*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'API violation - called -[XCTestExpectation fulfill] after the wait context has ended.'
*** First throw call stack:
(
  0   CoreFoundation   0x000000010c3a6f35 __exceptionPreprocess + 165
  1   libobjc.A.dylib  0x000000010a760bb7 objc_exception_throw + 45
  2   CoreFoundation   0x000000010c3a6d9a +[NSException raise:format:arguments:] + 106
  3   Foundation       0x000000010a37d5df -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 195
  4   XCTest           0x0000000115c48ee1 -[XCTestExpectation fulfill] + 264
  ...
)
libc++abi.dylib: terminating with uncaught exception of type NSException

Of course I can check if the test is finished before calling the fulfill method like this:

- (void) test1TaskLongerThanTimeout
{
    XCTestExpectation *expectation = [self expectationWithDescription:@"Test 1: task longer than timeout"];

    __block BOOL testIsFinished = NO;
    [self startAsynchronousTaskWithDuration:4 completionHandler:^(id result, NSError *error) {
        if (testIsFinished) {
            return;
        }
        XCTAssertNotNil(result);
        XCTAssertNil(error);
        [expectation fulfill];
    }];

    [self waitForExpectationsWithTimeout:2 handler:^(NSError *error) {
        testIsFinished = YES;
    }];
}

But this seems overly complicated and makes the test much harder to read. Am I missing something? Is there a simpler way to solve this problem?

Answer 1

Yes, there is a much simpler way to avoid this API violation issue: just declare your expectation variable as __weak. Although not clearly documented, the expectation will be released when the timeout expires. So if the task takes longer than the timeout, the expectation variable will be nil when the task completion handler is called. Thus the fulfill method will be called on nil, doing nothing.

- (void) test1TaskLongerThanTimeout
{
    __weak XCTestExpectation *expectation = [self expectationWithDescription:@"Test 1: task longer than timeout"];
    [self startAsynchronousTaskWithDuration:4 completionHandler:^(id result, NSError *error) {
        XCTAssertNotNil(result);
        XCTAssertNil(error);
        [expectation fulfill];
    }];
    [self waitForExpectationsWithTimeout:2 handler:nil];
}

Answer 2

I came across the same problem but in my case I needed the Swift version of the answer above.

I'm working on an OpenStack Swift Drive for OSX. When a folder is deleted locally with Finder, the deletion eventually propagates to the Server, I needed a test that waits for the server to be updated.

To avoid the API violation crash, I've changed my expectations to be "weak var" and changed the call to fulfill it to "zeroFoldersExpectation?.fulfill()" with the extra '?' as the expectation is now optional and could become nil in which case the fulfill call is ignored. This fixed the crashes.

func testDeleteFolder()
{
    Finder.deleteFolder()

    weak var zeroFoldersExpectation=expectationWithDescription("server has zero folders")
    Server.waitUntilNServerFolders(0, withPrefix: "JC/TestSwiftDrive/", completionHandler: {zeroFoldersExpectation?.fulfill()})
    waitForExpectationsWithTimeout(10, handler: {error in})

}

Answer 3

Instead of creating expectation as weak variable (as suggested in this answer) I think you could also set is as block variable and nil it in completion handler of waitForExpectationsWithTimeout:

- (void) test1TaskLongerThanTimeout
{
    __block XCTestExpectation *expectation = [self expectationWithDescription:@"Test 1: task longer than timeout"];
    [self startAsynchronousTaskWithDuration:4 completionHandler:^(id result, NSError *error) {
        XCTAssertNotNil(result);
        XCTAssertNil(error);
        [expectation fulfill];
    }];
    [self waitForExpectationsWithTimeout:2 handler:^(NSError *error) {
        expectation = nil;
    }];
}

This way you are sure that ARC won't dealloc expectation too fast.