Threadpool deadlock with Task.Result

Question

We have a large legacy with asp.net system, we have started using some async methods from an infrastructure library that we cannot change. The system doesn't use tasks in most places but the infrastructure exposes only async methods.

In the code we use the following pattern to use the async methods:

Task.Run(() => Foo()).Result

We use the Task.Run to prevent a deadlock if somewhere in the code someone didn't use ConfigureAwait(false), There are a lot of places that someone could have missed and it has happened before. And we use Task.Result to integrate it with the existing sync code base.

After experiencing heavy load We've noticed that that we are getting timeouts but the servers are not doing any work(Low CPU), we've found out that when there are a lot of calls to the server and the thread pool reaches the maximal number of threads the threads reach a deadlock in the Task.Result as it blocks the thread until the task is finished but the task can't run as there are no thread pool threads available to run it.

The best solution would be to change the code to work async all the way, but that is not an option right now. Also removing the Task.Run could work but it's too risky as there is not enough test coverage to know that we will not cause new deadlocks in untested flows.

I've tried to implement a new task scheduler that will not use the thread pool but a different set of threads to run the Foo task, but internal task are being executed on the default task scheduler that I do not want to replace.

Any ideas how this can be resolved without a huge change to the code base?

This is a small sample app that reproduces the issue, using only 10 threads and not the real limit. In the sample Foo will never be called.

class Program
{
    static void Main(string[] args)
    {
        ThreadPool.SetMaxThreads(10, 10);
        ThreadPool.SetMinThreads(10, 10);

        for (int i = 0; i < 10; i++)
        {
            ThreadPool.QueueUserWorkItem(CallBack);
        }

        Console.ReadKey();
    }

    private static void CallBack(object state)
    {
        Thread.Sleep(1000);

        var result = Task.Run(() => Foo()).Result;
    }

    public static async Task<string> Foo()
    {
        await Task.Delay(100);
        return "";
    }
}

Answer 1

You have explained your issue very well.

As you are using Task.Run then blocking on the result, you are using 1-2 threads, when really with async you want to use 0-1 threads.

If you are using Task.Run too liberally through your code then there is a chance you'll have multiple layers of blocking threads, making thread usage get really ugly fast, and you'll hit the maximum capacity as you've described.

As an aside, forget trying to find async deadlocks in unit tests (or console apps) as it requires as non-default SynchronizationContext.

The best and right solution would be to make everything top-to-bottom async or sync, but given you are constrained, I'd suggest investigating this wonderful library from the Microsoft vs team and look at JoinableTaskFactory.Run(...), this will run continuations on the blocking thread, and plays well when you nest this pattern at multiple levels. Using this approach, you will get closer to the synchronous equivalent code.

To reiterate, these techniques are workarounds, and if you are justifying these workarounds by respecting the existing code, the best way to respect it is to do it right, and make it fully sync, or top-to-bottom async.