NewRelic, async http handler and AcquireRequestState

Question

I have a problem with one async handler in distributed ASP.NET web app. First let me explain a use case:

• application uses IIS 8 on win 2012 machine with .NET Framework 4.5.2

• application has disabled Session and authentication modules via web.config like this

       <system.webServer>
         ....
         <modules>
              <remove name="WindowsAuthentication" />
              <remove name="Session" />
              <remove name="FormsAuthentication" />
          </modules>
       </system.webServer>
  

• application uses custom async web handler to serve the specific request

• application has very heavy traffic (about 50k requests per minute per server, async handler has about 10k requests per minute per server all tracked from NewRelic)
• application is distributed via multiple w3wp processes (2 w3wp processes) and multiple virtual servers (about 10 servers)
• application has high amount of connections

All normal (sync requests) are working fine but async request that does a little more work (that's why we use async request) is often slow but NewRelic reports that it is slow because of "AcquireRequestState". Now I've looked on google and stack overflow and this event is connected to creating a Session but we have sessions disabled in web.config. Does anyone know what else could "AcquireRequestState" could be doing? Are we missing some place to remove session state? Adding that from web.config to machine.config did nothing...

Here is a snippet from a request in NewRelic:

   **Slowest components   Count Duration     %   **
     AcquireRequestState    1   12,600 ms   100%  --> WTF?
     ExecuteRequestHandler  1   5.01 ms     0%
     Integrated Pipeline    1   0.334 ms    0%
     UpdateRequestCache     1   0.3 ms      0%
     EndRequest             1   0.168 ms    0%
     AuthenticateRequest    1   0.161 ms    0%
     Total time                 12,600 ms   100%

EDIT: I have <sessionState mode="Off" /> in web.config (<system.web> section) so that is not it.

Answer 1

I've looked into this because we had similar issues, I found this forum post, the reply that is interesting is this:

Unfortunately, the issue is not as simple as just turning sessionState off. In fact, one of the key phrases when describing the challenges with AcquireRequestState is the phrase for example, session state when it comes to when this event is raised. In digging deeper into this (actually looking at the .NET source) we can see that this is called when an EventHandler is executed or a RequestNotification object is created. I daresay there are other methods and/or events that, when called, will raise an AcquireRequestState event. Tracking down all of them represents something of a struggle. It seems this is something not talked about a lot outside of the more normalized session state discussions. The most common place we see this event raised is certainly related to session state management. But there are very obviously outliers where these event can still be raised. It can even be called directly from application code. The thing the agent grapples with is that it can identify the event, but rarely the source. When it is raised as part of the ASP pipeline, the only notification the agent gets is that this is one segment of the transaction. The source, or the methods executed inside of the event, is something the agent is rarely instrumenting by default. I wish we could offer more insight for you on this. There are a lot of moving parts inside of a .NET application, some of which involve the operating system itself, IIS, the version of .NET, whether or not the methods are asynchronous, application pool settings, execution modes, permissions, etc. While I don't want to open a second can of worms here, this harkens to the issue with the lack of stack traces for 500 errors. The agent provides a stack trace when it is offered and/or available. Where the stack trace, if one even exists, occurs within the ASP pipeline is extremely important. Sometimes it occurs before any actual application code is executed. In such cases the error is reported to the application, which in turn lets the .NET agent see and report that an error occurred, but no other details are provided. The agent simply sees that it happened and reports as much information as is possible. Beyond that the agent simply has no further details it can offer.

We gave up, so I'd be interested to know if you figure it out!