Sporadic application deadlock in ASP.NET Identity

Question

I have an ASP.NET 4.7.2 MVC 5 application that uses ASP.NET Identity and the OWIN OAuthAuthorizationServerMiddleware. In the RefreshTokenProvider.OnReceive method i'm accessing the SignInManager.CreateUserIdentity method which is an ASP.NET Identity method that internally uses AsyncHelper (see below) to call the asynchronous method. Every so often (typically months apart on a busy system), this breaks down and locks up the entire application. From a memory dump i've gathered, there were 565 threads waiting inside GetResult

// Copyright (c) Microsoft Corporation, Inc. All rights reserved.
// Licensed under the MIT License, Version 2.0. See License.txt in the project root for license information.

using System;
using System.Globalization;
using System.Threading;
using System.Threading.Tasks;

namespace Microsoft.AspNet.Identity
{
    internal static class AsyncHelper
    {
        private static readonly TaskFactory _myTaskFactory = new TaskFactory(CancellationToken.None,
            TaskCreationOptions.None, TaskContinuationOptions.None, TaskScheduler.Default);

        public static TResult RunSync<TResult>(Func<Task<TResult>> func)
        {
            var cultureUi = CultureInfo.CurrentUICulture;
            var culture = CultureInfo.CurrentCulture;
            return _myTaskFactory.StartNew(() =>
            {
                Thread.CurrentThread.CurrentCulture = culture;
                Thread.CurrentThread.CurrentUICulture = cultureUi;
                return func();
            }).Unwrap().GetAwaiter().GetResult();
        }

        public static void RunSync(Func<Task> func)
        {
            var cultureUi = CultureInfo.CurrentUICulture;
            var culture = CultureInfo.CurrentCulture;
            _myTaskFactory.StartNew(() =>
            {
                Thread.CurrentThread.CurrentCulture = culture;
                Thread.CurrentThread.CurrentUICulture = cultureUi;
                return func();
            }).Unwrap().GetAwaiter().GetResult();
        }
    }
}

The memory dump showed 580+ Tasks, that were all in the RanToCompletion state. I was unable to diagnose why GetResult does not succeed, given that the Task has completed, nor why that many threads pile up there rapidly when it worked for months before and why the entire app becomes unresponsive, even if they don't exercise this path. This caused a production outage now multiple times and i don't see how to resolve this other than rebooting it.

I've tried to use the OnReceiveAsync method but those seem pointless because before they're invoked, there's this little snippet:

if (OnReceiveAsync != null && OnReceive == null)
{
    throw new InvalidOperationException(Resources.Exception_AuthenticationTokenDoesNotProvideSyncMethods);
}

EDIT: Reproduction and explanation of the issue:

The issue can be reproduced by this WebAPI 2 Controller

using System.Threading.Tasks;
using System.Web.Http;

namespace WebApplication65.Controllers
{
    public class ValuesController : ApiController
    {
        public string Post([FromBody]string value)
        {
            return AsyncHelper.RunSync(PostAsync);
        }

        private async Task<string> PostAsync()
        {
            await Task.Delay(10);
            return "Hello World";
        }
    }
}

and this program to generate load

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;

namespace ConsoleApp36
{
    class Program
    {
        static void Main(string[] args)
        {
            HttpClient client = new HttpClient();
            MediaTypeHeaderValue mediaTypeHeaderValue = new MediaTypeHeaderValue("application/json");
            var tasks = new List<Task>();
            for (int i = 0; i < 100; i++)
            {
                int x = i;
                var t = Task.Run(async () =>
                {
                    var content = new StringContent("\"" + Guid.NewGuid().ToString() + "\"");
                    content.Headers.ContentType = mediaTypeHeaderValue;
                    await client.PostAsync("https://localhost:44371/api/values", content);
                    Console.WriteLine(x);
                });
                tasks.Add(t);
            }
            Task.WhenAll(tasks).Wait();
        }
    }
}

What the program will do, is hit the application with 100 requests in just a few milliseconds. This will lead to all previously free threads being stuck at AsyncHelper.RunSync with a lot more requests queued and no response being sent yet at all. The ThreadPool notices that it needs more threads but will add about only one thread per second which will immediately get stuck at AsyncHelper.RunSync trying to serve one of the queued requests. About a minute later, when the ThreadPool has expanded by about 100 additional threads to serve the 100 requests, all pending requests will respond in the blink of an eye and the application is responsive again.

The difference on my app is, that requests keep comming in, unlike the sample where only 100 requests are sent at once. This means my app can't recover from this scenario as the ThreadPool does not create new threads fast enough to keep up with incomming requests.

Creating an async copy of my ReceiveRefreshToken method and populating OnReceiveAsync in addition to OnReceive appears to avoid the problem enough to not get exhausted on the ThreadPool.

Answer 1

I think the application exhausted ThreadPool.

AsyncHelper uses TaskScheduler.Default which means execution on ThreadPool. It should work fine until all threads are blocked (e.g. many users and/or slow response from token endpoint) and there're no more free threads to continue. It would lead to deadlock (you can read more about it here). Since application has only one ThreadPool all stops if it is exhausted.

I would try to use OnReceiveAsync anyway and create user in async way from here. You will likely need a dummy OnReceive in order to avoid exception.

Alternatively you can try to extend size of the ThreadPool but it is only a short term solution not guaranteed to work all the time.