How to test thread-safety with XCTest

Question

Suppose we have a following class with mutable state:

class Machine {
   var state = 0
}

Now, let's say that there are some internal mechanisms that control the state. However, state change can occur on any thread or queue, so reading and writing to the state property must be performed in thread safe environment. To achieve that we will use simple sync(:_) method on dispatch_queue_t to synchronize access to the state variable. (Not the only way to do this, but that's one example)

Now, we can create create one private variable that holds the state value and another public variable with custom setters and getters that utilizes dispatch_sync(_:) method.

class Machine {
    private var internalState = 0

    var state: Int {
        get {
            var value: Int?
            dispatch_sync(dispatch_get_main_queue()) {
                value = self.internalState
            }
            return value!
        }

        set(newState) {
            dispatch_sync(dispatch_get_main_queue()) {
                self.internalState = newState
            }
        }
    }
}

state now has safe synchronized access from any queue or thread - it's thread safe.

Now here's the question.

How to test this behavior using XCTest?

Since class Machine can have a complex state machine we need to test how it performs in any environment:

• Test access to state from any queue or thread
• Test writing to state from any queue or thread

What are best approaches for testing this kind of behavior successfully?

Currently, I'm creating array of custom dispatch queues and array of defined states. Then I use dispatch_async method to change the state and test its value. That introduces new issues with XCTest execution because I need to track when all state mutations finish. That solution seems rather complex and unmaintainable.

What are the things I can do differently to achieve better testing?

Answer 1

There are two important moving parts when considering to test thread-safe code like this:

1. that the state accessor only runs in the context of a lock
2. that the locking mechanism is actually thread safe.

While the first one can be relatively testable by using mocking techniques, the latter one is difficult to test, mainly because validating that some code is thread-safe involves unit testing code from multiple threads accessing the thread-safe resource at the same time. And even this technique is not bullet proof, as we cannot fully control the execution order of the threads that we create from the unit tests, nor the allocated time per thread to make sure we catch all possible race conditions that can occur.

Considering the above, I'd recommend writing a small class/struct that provides the locking mechanism, and use it within the state accessors. Separating the responsibilities like this makes it easier to asses the correctitude of the locking mechanism via code review.

So, my recommendation would be to move the thread-safe code into a dedicated wrapper, and to use that wrapper from the Machine class:

/// A struct that just wraps a value and access it in a thread safe manner
public struct ThreadSafeBox<T> {
    private var _value: T

    /// Thread safe value, uses the main thread to synchronize the accesses
    public var value: T {
        get {
            if Thread.isMainThread { return _value }
            else { return DispatchQueue.main.sync { _value } }
        }
        set {
            if Thread.isMainThread { _value = newValue }
            else { DispatchQueue.main.sync { _value = newValue } }
        }
    }

    /// Initializes the box with the given value
    init(_ value: T) {
        _value = value
    }
}

The ThreadSafeBox code is relatively small and any design flaws can be spotted at code review time, so theoretically its thread safeness can be proven by code analysis. Once we prove the reliability of the ThreadSafeBox, then we have the guarantee that Machine is also thread safe in respect to its state property.

If you really want to test the property accessors, you could validate the fact that the get/set operations run only on the main thread, this should be enough to verify the thread safeness. Just note that the locking mechanism is something tied to the implementation details of that class, and unit testing implementation details has the disadvantage of tightly coupling the unit and the unit test. And that can lead to the need for updating the test if the implementation details change, which makes the test less reliable.