ASP.NET Sessions Corrupted Under Load

Question

We are seeing, under load, session data becoming corrupted or missing, but the sessions themselves remain present.

Our website is hosted on IIS 7 running ASP.Net 4.0 and using InProc session state in a web-farm with a total of 4 servers behind a Cisco ACE Appliance load balancer.

At this time the issue is random, we cannot reproduce the issue at will. This web application has been functioning correctly for the last seven months.

We realize Microsoft does not recommend using InProc with web farms, even if sticky sessions are being used.

We do have a lab environment, reasonably identical to our production environment, but are unable to reproduce there under substantial load (we use WAPT).

In our production environment, we have tried isolating just one server behind the load balancer in order to eliminate "server hopping" caused by the load balancer itself. The problem persists, however, even when running on one server. We do not see any AppPool or IIS recycling happening at all in production. As a standard practice, we do recycle the production app pools daily at 3am EST, and have enjoyed months of uptime on the OS.

What's being stored in session are a wide variety of objects, from simple types (integers and strings), to our entire shopping cart (a complex object graph), and even a user control (.ascx) instance. Due to the inability to easily serialize many of these objects, we are unable to make the switch to out-of-process session storage in a reasonable period of time.

Someone suggested trying to capture an HTTP session using Fiddler. The issue with running Fiddler is that we can't intentionally reproduce the issue ourselves. So, this leaves us unable to capture an HTTP trace of a failure event as it occurs. The trace logs from WAPT in our lab would likely provide the same data as Fiddler, but as I said, we can't reproduce it there.

I'd greatly appreciate any insights anyone may have...

Answer 1

Based on all of the information gathered here so far, I am going to answer with an educated guess as to what the problem is.

The session is likely expiring somehow.

The app pool recycling is not the only thing that can cause a session expiry. And, as Hanselman will tell you, this is a common occurence when you combine InProc session management and high volume.

Edit: Take a look at an older blog post for IIS6 that details how to determine the cause of lost sessions like this, especially ones that affect specific users, not just all sessions, as that can happen, too. The section of interest is just after the Application_End code snippet talking about Web Gardens. I've looked for a newer piece of information similar to this and all I could really find that talked about all of the issues was an answer to another question here on SO.