Upgrade from .NET 3.0 to 3.5: Sites set to StateServer revert to InProc when in Web Garden

Question

Scenario:

Take a server running .NET 3.0 and an ASP.NET Web site running in an application pool that has Web gardens enabled (number of processes: 3). The web.config configuration is as follows:

    <sessionState
      cookieless="UseCookies"
      cookieName=".authz"
      mode="StateServer"
      regenerateExpiredSessionId="true"
      stateConnectionString="tcpip=127.0.0.1:42424"
      timeout="60"
      useHostingIdentity="true" />

Now upgrade the machine to .NET 3.5 SP1. Reboot the server. Result: sessions are no longer maintained across instances of w3wp.exe, as if all of the have reverted to InProc. Reducing to 1 worker process is the current workaround.

What's strange: Same code on different server experiences no problems. I've experienced this problem before, but it magically went away after a restart. I restarted once already, but no joy so far.

Comparing the two machine.configs and web.configs of the two servers: identical.

Someone else has experienced this problem, but no answer there.

Any ideas? I'm really stumped on this one.

Answer 1

So, this one was just amazing.

The problem appears to occur when all of the following conditions are true:

• You are running Windows Server 2003 (IIS 6.0) and an ASP.NET 2.0 web site.
• The Web site is configure to use Web Gardens, where the maximum number of worker processes is greater than 1. Because of this, you have configured your application to use an out-of-process session storage; in this scenario, the ASP.NET State Service running on the local machine.
• The application pool identity is set not to NETWORK SERVICE but to a custom, low-privileged user account that you created per a deployment best practice.
• You run an installer that updates the .NET framework; in my case, this was an update from .NET 3.0 to .NET 3.5 SP1.

When the upgrade finishes and you reboot the server, you find that your session variables are frequently lost upon refreshing a page, since there is only a 1 in 3 chance of you getting the original worker process that served your original request. But this shouldn't matter, since you're using the ASP.NET state service. What broke?

When using the ASP.NET state service, ASP.NET uses a value called the machineKey to encrypt and/or hash all session data to be stored (I don't know if it's encrypting or hashing or both, but it's not an important distinction for this discussion). This is so that when any worker process asks for data from the service using a session identifier, it can be sure that the data was not tampered with while it was being stored in the external data source.

If you are on a web farm, then you probably have a static machineKey defined in your web.config file, and this issue does not occur. But for a single-server web garden scenario, you probably rely on the default machineKey setting, which is set to AutoGenerate,IsolateApps for ASP.NET 2.0 applications. This means that ASP.NET automatically generates a machine key that is unique to your application pool. It regenerates this key according to some algorithm, but that is not important for this discussion.

The generated value is normally stored in the registry under HKLM\SOFTWARE\Microsoft\ASP.NET\2.0.50727.0\AutoGenKeys\{SID of the Application Pool Identity}. But the .NET Framework installer incorrectly (I do believe this is a bug) destroys this registry key and, to add insult to injury, resets the permissions on this key such that your custom application pool identity cannot write to the registry entry when it goes to create its new machine key.

The result is that each worker process that spins up in the web garden is using its own in-memory copy of a machine key that it generated just in time, effectively creating a web farm scenario by accident. For example, worker process A spins up, sees that no AutoGenKey entry exists (indeed, it cannot even read it), generates its own and begins using that to hash data sent to the ASP.NET State Service. It tries to save this new machine key to the registry entry, but fails silently. Worker process B spins up, sees that no AutoGenKey entry exists, generates its own and begins using that to hash data...you see where this is going.

The result is now you have session data hashed with three different machine keys. Though the data for the session identifier exists, two out of three of the worker processes will reject it as invalid/tampered because it is using its own key.

You could get around this by explicitly setting a custom machineKey in your web.config file.

Or you could re-run aspnet_regiis.exe -ga MachineName\ApplicationPoolUserName at a Command Prompt to fix up the broken permissions.

Your problem is solved. Time to go to bed.

---

UPDATE June 30th: Per my report of this issue on Microsoft Connect, Microsoft has indicated that they have fixed the installer such that this behavior won't happen beginning with upgrades to .NET 4. It still could happen for all future 3.0/3.5 upgrades, so I'll leave this question/answer standing.