Asp.net randomly stops obeying forms authentication whitelist

Question

The problem

Last month we move our asp.net website farm from Server 2008 R2 to Server 2012 R2 and upgraded to asp.net 4.5. We are using cookied forms authentication to prevent unauthorized access to the website.

<authorization>
  <deny users="?" />
  <allow users="*" />
</authorization>

We have certain assets and pages (ex: sign in page) that are whitelisted in the web.config:

<location path="signin">
   <system.web>
     <authorization>
       <allow users="*" />
     </authorization>
   </system.web>
</location>

Over the last few months we've been noticing that IIS/Asp.net randomly stops obeying the whitelist and assume everything needs to be authenticated. All requests to the site on that server will be redirected to the signin page which then throws a 500 error. No whitelisted assets can be retrieved.

There are then 2 errors in the event viewer that we can see when IIS is messed up. The first:

Exception type: NullReferenceException 
    Exception message: Object reference not set to an instance of an object.
   at System.Web.PipelineModuleStepContainer.GetNextEvent(RequestNotification notification, Boolean isPostEvent, Int32 eventIndex)
   at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
   at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)

This second one doesn't show up all the time:

Event code: 4005 
Event message: Forms authentication failed for the request. Reason: The ticket supplied has expired. 

The iis process will be working find for hours then all the sudden start doing this weirdness. As soon as we recycle the app pool, or even just modify the web.config the site starts working again.

Things we've tried

Honestly we are quite stumped. This wasn't happening on our old servers, but we've made quite a few changes to our site since then but nothing related to authentication.

1. We are in a webfarm and we define our machine key inside of our web.config.

   <machineKey validationKey="XXX" decryptionKey="XXX" validation="SHA1" decryption="AES" />
   

2. We are targeting asp.net 4.5

   <httpRuntime targetFramework="4.5" executionTimeout="120" maxQueryStringLength="4096"  minFreeThreads="72" minLocalRequestFreeThreads="88"  maxRequestLength="32768" />
   

3. We recreated the application pool within IIS.

4. Not sure if it matters but we use IIS Shared Config and shared certificates.
5. The issue is happening on all of the web servers in the farm, not just one.
6. We reinstalled the OS on one of the servers yesterday... so we'll see if that fixes anything.
7. It doesn't seem to be tied to memory usage. Sometimes iis is only using 4gb, sometimes 6gb.
8. It doesn't seem to be tied to a certain page execution that we can tell.
9. I've run debug diag against a memory dump and there aren't any threads that are running long nor crazy memory usage.

Yea, we are stumped. Any help is appreciated.

Answer 1

I had a similar experience, changing IIS to allow anonymous authentication solved it for me. In your case, I would recommend 2 things:

1. Try Enabling Anonymous Authentication in IIS (Visit http://technet.microsoft.com/en-us/library/cc770966%28v=ws.10%29.aspx to see how)
2. Modifiy your code as shown below

<location path="signin">
  <system.web>
    <authorization>
      <allow users="?" />
      <allow users="*" />
    </authorization>
  </system.web>
</location>

I hope this helps

Answer 2

I'll answer my own question with what we did to solve the issue, even though we never did find the root cause. We noticed that the server would start ignoring the whitelist rules when it got too heavy under load. Not much load, maybe 40% utilization over the course of 5 minutes. After that it would start ignoring things.

The simple solution for us was to throw more hardware at the issue. We are running 6 webservers instead of 3. We haven't seen the whitelist issue since then. So honestly... we have no idea what's up.