How do I fix the deadlock on a threadpool thread that has a SynchronizationContext?

Question

I am getting intermittent deadlocks when using HttpClient to send http requests and sometimes they are never returning back to await SendAsync in my code. I was able to figure out the thread handling the request internally in HttpClient/HttpClientHandler for some reason has a SynchronizationContext during the times it is deadlocking. I would like to figure out how the thread getting used ends up with a SynchronizationContext, when normally they don't have one. I would assume that whatever object is causing this SynchronizationContext to be set is also blocking on the Thread, which is causing the deadlock.

Would I be able to see anything relevant in the TPL ETW events?

How can I troubleshoot this?

Edit 2: The place that I have been noticing these deadlocks is in a wcf ServiceContract(see code below) inside of a windows service. The SynchronizationContext that is causing an issue is actually a WindowsFormsSynchronizationContext, which I assume is caused by some control getting created and not cleaned up properly (or something similar). I realize there almost certainly shouldn't be any windows forms stuff going on inside of a windows service, and I'm not saying I agree with how it's being used. However, I didn't write any of the code using it, and I can't just trivially go change all of the references.

Edit: here is an example of the general idea of the wcf service I was having a problem with. It's a simplified version, not the exact code:

[ServiceContract]
[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple)]
internal class SampleWcfService
{
    private readonly HttpMessageInvoker _invoker;

    public SampleWcfService(HttpMessageInvoker invoker)
    {
        _invoker = invoker;
    }
    
    [WebGet(UriTemplate = "*")]
    [OperationContract(AsyncPattern = true)]
    public async Task<Message> GetAsync()
    {
        var context = WebOperationContext.Current;
        using (var request = CreateNewRequestFromContext(context))
        {
            var response = await _invoker.SendAsync(request, CancellationToken.None).ConfigureAwait(false);
            var stream = response.Content != null ? await response.Content.ReadAsStreamAsync().ConfigureAwait(false) : null;
            return StreamMessageHelper.CreateMessage(MessageVersion.None, "GETRESPONSE", stream ?? new MemoryStream());
        }
    }
}

Adding ConfigureAwait(false) to the 2 places above didn't completely fix my problem because a threadpool thread used to service a wcf request coming into here may already have a SynchronizationContext. In that case the request makes it all the way through this whole GetAsync method and returns. However, it still ends up deadlocked in System.ServiceModel.Dispatcher.TaskMethodInvoker, because in that microsoft code, it doesn't use ConfigureAwait(false) and I want to assume there is a good reason for that (for reference):

var returnValueTask = returnValue as Task;

if (returnValueTask != null)
{
    // Only return once the task has completed                        
    await returnValueTask;
}

It feels really wrong, but would converting this to using APM (Begin/End) instead of using Tasks fix this? Or, is the only fix to just correct the code that is not cleaning up its SynchronizationContext properly?

Answer 1

Update: we now know we're dealing with a WindowsFormsSynchronizationContext (see comments), for whatever reason in a WCF application. It's no surprise then to see deadlocks since the point of that SyncContext is to run all continuations on the same thread.

You could try to to set WindowsFormsSynchronizationContext.AutoInstall to false. According to its docs, what it does is:

Gets or sets a value indicating whether the WindowsFormsSynchronizationContext is installed when a control is created

Assuming someone creates a WindowsForms control somewhere in your app, then that might be your issue and would potentially be solved by disabling this setting.

An alternative to get rid of an existing SynchronizationContext would be to just overwrite it with null, and later restoring it (if you're nice). This article describes this approach and provides a convenient SynchronizationContextRemover implementation you could use.

However, this probably won't work if the SyncContext is created by some library methods you use. I'm not aware of a way to prevent a SyncContext from being overwritten, so setting a dummy context won't help either.

---

Are you sure the SynchronizationContext is actually at fault here?

From this MSDN magazine article:

Default (ThreadPool) SynchronizationContext (mscorlib.dll: System.Threading)
The default SynchronizationContext is a default-constructed SynchronizationContext object. By convention, if a thread’s current SynchronizationContext is null, then it implicitly has a default SynchronizationContext.

The default SynchronizationContext queues its asynchronous delegates to the ThreadPool but executes its synchronous delegates directly on the calling thread. Therefore, its context covers all ThreadPool threads as well as any thread that calls Send. The context “borrows” threads that call Send, bringing them into its context until the delegate completes. In this sense, the default context may include any thread in the process.

The default SynchronizationContext is applied to ThreadPool threads unless the code is hosted by ASP.NET. The default SynchronizationContext is also implicitly applied to explicit child threads (instances of the Thread class) unless the child thread sets its own SynchronizationContext.

If the SynchronizationContext you are seeing is the default one, it should be fine (or rather, you will have a very hard time to avoid it being used).

Can't you provide more details / code about what's involved?

One thing that looks immediately suspicious to me in your code (though it may be completely fine) is that you have a using block that captures a static WebOperationContext.Current in request, which will both be captured by the generated async state machine. Again, might be fine, but there's a lot of potential for deadlocks here if something waits on WebOperationContext

Answer 2

Try below; I have found success in similar cases getting into the async rabbit hole.

var responsebytes = await response.Content.ReadAsByteArrayAsync();
MemoryStream stream = new MemoryStream(filebytes);

Response the stream variable.

Hope it helps.