To limit concurrency OR NOT to limit concurrency? (within a single ASP.NET request)

Question

There are plenty of answers on HOW to limit concurrency of async I/O operations and/or their continuations -- using a custom scheduler, SemaphorSlim etc. My question is: does it make sense to do that in a standard ASP.NET MVC / WebAPI scenario?

We have a typical enterprise API that serves as backend for a customer-facing SPA. Many API requests involve calling dozens of downstream web services, which we by now have mostly converted to async I/O with TAP (async/await). Many of these remote service calls are started in parallel, without awaiting, and then awaited in bulk using Task.WhenAll. Sometimes WhenAll is done over a fixed number of tasks, and sometimes we start a remote call per item in collection - which leads to spawning an unknown (but usually low, under a dozen) number of tasks and then awaiting them with WhenAll.

As I understand, this causes continuations of these tasks (i.e. the logic that deserializes their responses) to get scheduled on the ThreadPool. Which leads to my question: would running these CPU-bound continuations in parallel not lead to excessive pressure on the ThreadPool? Should we develop some sort of middleware that would limit concurrency of the continuations of the async I/O tasks started inside a single request by scheduling them to a custom scheduler?

Would that allow for better scalability of our app, by reducing the number of ThreadPool threads allocated by any given request, which would allow for more concurrent requests to be served before we'd start running out of ThreadPool threads (or getting throttled by the ThreadPool growth)?

Or is this rather useless, and we should simply trust the default ThreadScheduler+ThreadPool's ability to schedule all tasks to the available CPU cores across whatever number of concurrent requests?

FYI to everyone attempting to answer:

This is a rather mature system at a very large company, well-scrutinized by dozens of experts (including myself), in production for one US state, about to enter production at the national level. Suggestions like "measure first", "know whether you are IO vs. CPU bound", "try AppInsights" and "don't try to be smarter than Microsoft" are the very first things we ourselves have obviously thought about. The level of guidance we're seeking is more like: has anyone here implemented an all-async ASP.NET Web API system at the US national level and has real-life experience with async/WhenAll concurrency?

Answer 1

@tarun-lalwani make some great points in the comments

But I'll mention the solid advice to never preoptimize. A mid-range desktop today can easily handle a threadpool of 16 (which can handle hundreds of waiting tasks generally), a good server can handle a much larger thread pool than that, how much does your time cost vs the cost of a replacement or new server, probably less than 6 months, very possibly less than 3 months.

Any performance problem comes down to the bottlenecks, the system can only perform as fast as the slowest bottleneck. So improving those specific bottlenecks, in the real-world system, is how to improve performance. How to scale those groups of bottlenecks is how to improve scalability. There are still a lot of differences between desktop and server hardware, so I'll stress again that you need to see real-world not in-vitro metrics on these things before putting in weeks/months of effort into fixing a "problem" that may not be there.

GraphQL as mentioned allows you to rearrange where the bottlenecks occur, allowing you to distribute them more evenly, to better utilize the hardware you already have. Which is also true of many of the more modern design patterns.

In summary, it only makes sense to limit a process that if and when it becomes a problem.