Deadlock when combining app domain remoting and tasks

Question

My app needs to load plugins into separate app domains and then execute some code inside of them asynchronously. I've written some code to wrap Task in marshallable types:

static class RemoteTask
{
    public static async Task<T> ClientComplete<T>(RemoteTask<T> remoteTask,
                                                  CancellationToken cancellationToken)
    {
        T result;

        using (cancellationToken.Register(remoteTask.Cancel))
        {
            RemoteTaskCompletionSource<T> tcs = new RemoteTaskCompletionSource<T>();
            remoteTask.Complete(tcs);
            result = await tcs.Task;
        }

        await Task.Yield(); // HACK!!

        return result;
    }

    public static RemoteTask<T> ServerStart<T>(Func<CancellationToken, Task<T>> func)
    {
        return new RemoteTask<T>(func);
    }
}

class RemoteTask<T> : MarshalByRefObject
{
    readonly CancellationTokenSource cts = new CancellationTokenSource();
    readonly Task<T> task;

    internal RemoteTask(Func<CancellationToken, Task<T>> starter)
    {
        this.task = starter(cts.Token);
    }

    internal void Complete(RemoteTaskCompletionSource<T> tcs)
    {
        task.ContinueWith(t =>
        {
            if (t.IsFaulted)
            {
                tcs.TrySetException(t.Exception);
            }
            else if (t.IsCanceled)
            {
                tcs.TrySetCancelled();
            }
            else
            {
                tcs.TrySetResult(t.Result);
            }
        }, TaskContinuationOptions.ExecuteSynchronously);
    }

    internal void Cancel()
    {
        cts.Cancel();
    }
}

class RemoteTaskCompletionSource<T> : MarshalByRefObject
{
    readonly TaskCompletionSource<T> tcs = new TaskCompletionSource<T>();

    public bool TrySetResult(T result) { return tcs.TrySetResult(result); }
    public bool TrySetCancelled() { return tcs.TrySetCanceled(); }
    public bool TrySetException(Exception ex) { return tcs.TrySetException(ex); }

    public Task<T> Task
    {
        get
        {
            return tcs.Task;
        }
    }
}

It's used like:

sealed class ControllerAppDomain
{
    PluginAppDomain plugin;

    public Task<int> SomethingAsync()
    {
        return RemoteTask.ClientComplete(plugin.SomethingAsync(), CancellationToken.None);
    }
}

sealed class PluginAppDomain : MarshalByRefObject
{
    public RemoteTask<int> SomethingAsync()
    {
        return RemoteTask.ServerStart(async cts =>
        {
            cts.ThrowIfCancellationRequested();
            return 1;
        });
    }
}

But I've run into a snag. If you look in ClientComplete, there's a Task.Yield() I've inserted. If I comment this line, ClientComplete will never return. Any ideas?

Answer 1

My best guess is that you are facing these issues because of the async method that contains await and this is managed via the ThreadPool which can allocate some recycled Thread.

Reference Best practice to call ConfigureAwait for all server-side code

Actually, just doing an await can do that(put you on a different thread). Once your async method hits an await, the method is blocked but the thread returns to the thread pool. When the method is ready to continue, any thread is snatched from the thread pool and used to resume the method.

Try to streamline the code, generate threads for baseline cases and performance is last.