IIS ASP.NET WebApi Deadlock when requesting the same server

Question

We've been experiencing some deadlocks when working with interconnected ASP.NET WebApis on the same IIS server. We'd like to know if this is somehow an expected behavior, due to hosting all APIs on the same server and same Application Pool, since we have managed to avoid the issue by moving either WebApi to a different pool; or if there's something wrong with our code.

For production, we will probably host the APIs on different server or pools, but still we'd like to understand why this is happening. Our main concern is that, if it's our faulty code, the issue may be reproduced on a larger scale, even if the hosting setup is correct.

We have created a little solution to reproduce the deadlock, hosted in GitHub.

The reproduction steps are as follow:

1. WebClient executes multiple HTTP request in parallel WebApi1.
2. WebApi1 executes an HTTP request to WebApi2.
3. WebApi2 executes an HTTP request to WebApi3.
4. WebApi3 simply returns a string.

The expected behavior would be that all requests are eventually resolved.

The actual behavior is that, certain requests gets completed, while some others will fail, due to a TaskCancelledException which seems to be due to the requests timing out.

The only article that I was able to find that seems to mention the same issue is from 2014: "Do Not Send ServerXMLHTTP or WinHTTP Requests to the Same Server", I believe that this is the issue we are experiencing, how can we confirm this?

Context

We've been assigned the task to create a centralized authentication server for multiple internal APIs of the company we work at. We are using IdentityServer3 with reference tokens, so when some API requests a second API using reference tokens, the second API will request the authentication server for token validation which reproduces the issue.

I have added the IdentityServer tag, since this could be a common issue when doing multiple APIs communication and using reference tokens. Sample on GitHub.

Answer 1

Just one observation: you are using HttpClient as a static member for every Controller and according to this HttpClient is not guaranteed to be thread-safe