Profiler BLOCKED_TIME in IdentityServer4/Newtonsoft.Json

Question

I'm having issues that the /connect/introspect endpoint of my IdentityServer is sometimes really slow (10 seconds for one call). As you can see below, most of the calls (18k) perform quickly (<250ms).

I've enabled the new Application Insights profiling and most of the slow traces look like this:

As said on the Application Insights profiler page:

BLOCKED_TIME indicates the code is waiting for another resource to be available, such as waiting for a synchronization object, waiting for a thread to be available, or waiting for a request to finish.

But I have no reason to believe this should be waiting for something. There is no spike in requests so I don't think there aren't enought threads available or something. Memory seems to be no issue for our app service plan because it's always around 40%. I've also dug into the source of IdentityServer4 but couldn't identify any cause for this. So now I'm kind of stuck. Can anyone point me to possible causes for these slow requests? Or point me in the good direction to determine a cause? Any help will be much appreciated!

Edit with some extra info: We use IdentityServer4.EntityFramework to store reference tokens in a sql azure db. I've looked in Application Insights and the queries in those slow requests perform in less then 50ms. So I'm guessing it's not the database.

Answer 1

Look at the info, All of your requests are around 3222 ms, up until you start serialize your JSON. When you start serialize your data to json JSON the request jumps up to 5589 ms, when you deserialize it it jumps to 8811ms.

The issue here is not the DB, the DB might get the data fast enough. Not the spike in the request, which you don't have, nor the memory problem that doesn't exist.

The issue is the fact that you are fetching a lot of data, presumably and the compiler has it's penalty when serializing and deserializing the data, both of this actions are pretty costly.

You have to arrange your list as a JSON, then deserialize it into an object that you can access again.

Look at what happens between those calls, the amount of data you are serializing and deserializing and what happens after.

That is your key.