How can I wait for an async function from synchronous function in Swift 5.5?

Question

When conforming to protocols or overriding superclass methods, you may not be able to change a method to be async, but you may still want to call some async code. For example, as I am rewriting a program to be written in terms of Swift's new structured concurrency, I would like to call some async set-up code at the beginning of my test suite by overriding the class func setUp() defined on XCTestCase. I want my set-up code to complete before any of the tests run, so using Task.detached or async { ... } is inappropriate.

Initially, I wrote a solution like this:

final class MyTests: XCTestCase {
    override class func setUp() {
        super.setUp()
        unsafeWaitFor {
            try! await doSomeSetup()
        }
    }
}

func unsafeWaitFor(_ f: @escaping () async -> ()) {
    let sema = DispatchSemaphore(value: 0)
    async {
        await f()
        sema.signal()
    }
    sema.wait()
}

This seems to work well enough. However, in Swift concurrency: Behind the scenes, runtime engineer Rokhini Prabhu states that

Primitives like semaphores and condition variables are unsafe to use with Swift concurrency. This is because they hide dependency information from the Swift runtime, but introduce a dependency in execution in your code... This violates the runtime contract of forward progress for threads.

She also includes a code snippet of such an unsafe code pattern

func updateDatabase(_ asyncUpdateDatabase: @Sendable @escaping () async -> Void) {
    let semaphore = DispatchSemaphore(value: 0)

    async {
        await asyncUpdateDatabase()
        semaphore.signal()
    }

    semaphore.wait()

}

which is notably the exact pattern I had come up with (I find it very amusing that the code I came up with is exactly the canonical incorrect code modulo renaming).

Unfortunately, I have not been able to find any other way to wait for async code to complete from a synchronous function. Further, I have not found any way whatsoever to get the return value of an async function in a synchronous function. The only solutions I have been able to find for this on the internet seem just as incorrect as mine, for example this The Swift Dev article says that

In order to call an async method inside a sync method, you have to use the new detach function and you still have to wait for the async functions to complete using the dispatch APIs.

which I believe to be incorrect or at least unsafe.

What is a correct, safe way to wait for an async function from a synchronous function to work with existing synchronous class or protocol requirements, unspecific to testing or XCTest? Alternatively, where can I find documentation spelling out the interactions between async/await in Swift and existing synchronization primitives like DispatchSemaphore? Are they never safe, or can I use them in special circumstances?

Update:

As per @TallChuck's answer which noticed that setUp() always runs on the main thread, I have discovered that I can intentionally deadlock my program by calling any @MainActor function. This is excellent evidence that my workaround should be replaced ASAP.

Explicitly, here is a test which hangs.

import XCTest
@testable import Test

final class TestTests: XCTestCase {
    func testExample() throws {}
    
    override class func setUp() {
        super.setUp()
        unsafeWaitFor {
            try! await doSomeSetup()
        }
    }
}

func doSomeSetup() async throws {
    print("Starting setup...")
    await doSomeSubWork()
    print("Finished setup!")
}

@MainActor
func doSomeSubWork() {
    print("Doing work...")
}

func unsafeWaitFor(_ f: @escaping () async -> ()) {
    let sema = DispatchSemaphore(value: 0)
    async {
        await f()
        sema.signal()
    }
    sema.wait()
}

However, it does not hang if @MainActor is commented out. One of my fears is that if I ever call out to library code (Apple's or otherwise), there is no way to know if it will eventually call an @MainActor function even if the function itself is not marked @MainActor.

My second fear is that even if there is no @MainActor, I still don't know I am guaranteed that this is safe. On my computer, this hangs.

import XCTest
@testable import Test

final class TestTests: XCTestCase {
    func testExample() throws {}
    
    override class func setUp() {
        super.setUp()
        unsafeWaitFor {
            unsafeWaitFor {
                unsafeWaitFor {
                    unsafeWaitFor {
                        unsafeWaitFor {
                            unsafeWaitFor {
                                print("Hello")
                            }
                        }
                    }
                }
            }
        }
    }
}
func unsafeWaitFor(_ f: @escaping () async -> ()) {
    let sema = DispatchSemaphore(value: 0)
    async {
        await f()
        sema.signal()
    }
    sema.wait()
}

If this doesn't hang for you, try adding more unsafeWaitFors. My development VM has 5 cores, and this is 6 unsafeWaitFors. 5 works fine for me. This is distinctly unlike GCD. Here is an equivalent in GCD which does not hang on my machine.

final class TestTests: XCTestCase {
    func testExample() throws {}
    
    override class func setUp() {
        super.setUp()
        safeWaitFor { callback in
            safeWaitFor { callback in
                safeWaitFor { callback in
                    safeWaitFor { callback in
                        safeWaitFor { callback in
                            safeWaitFor { callback in
                                print("Hello")
                                callback()
                            }
                            callback()
                        }
                        callback()
                    }
                    callback()
                }
                callback()
            }
            callback()
        }
    }
}
func safeWaitFor(_ f: @escaping (() -> ()) -> ()) {
    let sema = DispatchSemaphore(value: 0)
    DispatchQueue(label: UUID().uuidString).async {
        f({ sema.signal() })
    }
    sema.wait()
}

This is fine because GCD is happy to spawn more threads than you have CPUs. So maybe the advice is "only use as many unsafeWaitFors as you have CPUs", but if that's the case, I would like to see somewhere that Apple has spelled this out explicitly. In a more complex program, can I actually be sure that my code has access to all the cores on the machine, or is it possible that some other part of my program is using the other cores and thus that the work requested by unsafeWaitFor will never be scheduled?

Of course, the example in my question is about tests, and so in that case, it is easy to say "it doesn't really matter what the advice is: if it works, it works, and if it doesn't, the test fails, and you'll fix it," but my question isn't just about tests; that was just an example.

With GCD, I have felt confident in my ability to synchronize asynchronous code with semaphores (on my own DispatchQueues that I control, and not the main thread) without exhausting the total available threads. I would like be able to synchronize async code from a synchronous function with async/await in Swift 5.5.

If something like this is not possible, I would also accept documentation from Apple spelling out in exactly what cases I can safely use unsafeWaitFor or similar synchronization techniques.

Answer 1

You could maybe argue that asynchronous code doesn't belong in setUp(), but it seems to me that to do so would be to conflate synchronicity with sequential...icity? The point of setUp() is to run before anything else begins running, but that doesn't mean it has to be written synchronously, only that everything else needs to view it as a dependency.

Fortunately, Swift 5.5 introduces a new way of handling dependencies between blocks of code. It's called the await keyword (maybe you've heard of it). The most confusing thing about async/await (in my opinion) is the double-sided chicken-and-egg problem it creates, which is not really addressed very well in any materials I've been able to find. On the one hand, you can only run asynchronous code (i.e. use await) from within code that is already asynchronous, and on the other hand, asynchronous code seems to be defined as anything that uses await (i.e. runs other asynchronous code).

At the lowest level, there must eventually be an async function that actually does something asynchronous. Conceptually, it probably looks something like this (note that, though written in the form of Swift code, this is strictly pseudocode):

func read(from socket: NonBlockingSocket) async -> Data {
    while !socket.readable {
        yieldToScheduler()
    }

    return socket.read()
}

In other words, contrary to the chicken-and-egg definition, this asynchronous function is not defined by the use of an await statement. It will loop until data is available, but it allows itself to be preempted while it waits.

At the highest level, we need to be able to spin up asynchronous code without waiting for it to terminate. Every system begins as a single thread, and must go through some kind of bootstrapping process to spawn any necessary worker threads. In most applications, whether on a desktop, smart phone, web server, or what have you, the main thread then enters some kind of "infinite" loop where it, maybe, handles user events, or listens for incoming network connections, and then interacts with the workers in an appropriate way. In some situations, however, a program is meant to run to completion, meaning that the main thread needs to oversee the successful completion of each worker. With traditional threads, such as the POSIX pthread library, the main thread calls pthread_join() for a certain thread, which will not return until that thread terminates. With Swift concurrency you..... can't do anything like that (as far as I know).

The structured concurrency proposal allows top-level code to call async functions, either by direct use of the await keyword, or by marking a class with @main, and defining a static func main() async member function. In both cases, this seems to imply that the runtime creates a "main" thread, spins up your top-level code as a worker, and then calls some sort of join() function to wait for it to finish.

As demonstrated in your code snippet, Swift does provide some standard library functions that allow synchronous code to create Tasks. Tasks are the building block of the Swift concurrency model. The WWDC presentation you cited explains that the runtime is intended to create exactly as many worker threads as there are CPU cores. Later, however, they show the below image, and explain that a context switch is required any time the main thread needs to run.

As I understand it, the mapping of threads to CPU cores only applies to the "Cooperative thread pool", meaning that if your CPU has 4 cores, there will actually be 5 threads total. The main thread is meant to remain mostly blocked, so the only context switches will be the rare occasions when the main thread wakes up.

It is important to understand that under this task-based model, it is the runtime, not the operating system, that controls "continuation" switches (not the same as context switches). Semaphores, on the other hand, operate at the operating system level, and are not visible to the runtime. If you try to use semaphores to communicate between two tasks, it can cause the operating system to block one of your threads. Since the runtime cannot track this, it will not spin up a new thread to take its place, so you will end up under-utilized at best, and deadlocked at worst.

Okay, finally, in Meet async/await in Swift, it is explained that the XCTest library can run asynchronous code "out of the box". However, it is not clear whether this applies to setUp(), or only to the individual test case functions. If it turns out that it does support an asynchronous setUp() function, then your question is suddenly entirely uninteresting. On the other hand, if it does not support it, then you are stuck in the position that you cannot directly wait on your async function, but that it's also not good enough just to spin up an unstructured Task (i.e. a task that you fire and forget).

Your solution (which I see as a workaround -- the proper solution would be for XCTest to support an async setUp()), blocks only the main thread, and should therefore be safe to use.