NullReferenceException at System.Web.Mvc.FilterProviderCollection.GetFilters

Question

I've an MVC 5 app (.NET 4.5.1) running on Windows 2008 R2 (IIS 7.5)

The following exception keeps being thrown periodically when running load tests. Unfortunately, I can't reproduce locally and am rather stuck, so am hoping the community may have more ideas. (Update: have now been able to reproduce under load)

Looking at the source for FilterProviderCollection.GetFilters suggests that its possibly the dependency resolver - but without more information, I'm reluctant to simply replace the library. Currently its using SimpleInjector.

If this is the case, my guess would be its caused by app pool recycles, but haven't been able to confirm this (enabling logging of all causes for a recycle in the app pool gave me nothing useful)

After much searching, I did find a few references to it potentially being Glimpse. I've confirmed this isn't the case. I've also stripped and rebuilt the project to help gain confidence that its not simply nuget-package upgrade weirdness.

Any suggestions as to what could be causing it, or how I may add extra logging to catch more information would be appreciated. Thanks.

Exception information: 
    Exception type: NullReferenceException 
    Exception message: Object reference not set to an instance of an object.
   at System.Web.Mvc.FilterProviderCollection.GetFilters(ControllerContext controllerContext, ActionDescriptor actionDescriptor)
   at System.Web.Mvc.ControllerActionInvoker.GetFilters(ControllerContext controllerContext, ActionDescriptor actionDescriptor)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.BeginInvokeAction(ControllerContext controllerContext, String actionName, AsyncCallback callback, Object state)
   at System.Web.Mvc.Controller.<BeginExecuteCore>b__1c(AsyncCallback asyncCallback, Object asyncState, ExecuteCoreState innerState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallBeginDelegate(AsyncCallback callback, Object callbackState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)
   at System.Web.Mvc.Async.AsyncResultWrapper.Begin[TState](AsyncCallback callback, Object callbackState, BeginInvokeDelegate`1 beginDelegate, EndInvokeVoidDelegate`1 endDelegate, TState invokeState, Object tag, Int32 timeout, SynchronizationContext callbackSyncContext)
   at System.Web.Mvc.Controller.BeginExecuteCore(AsyncCallback callback, Object state)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)
   at System.Web.Mvc.Async.AsyncResultWrapper.Begin[TState](AsyncCallback callback, Object callbackState, BeginInvokeDelegate`1 beginDelegate, EndInvokeVoidDelegate`1 endDelegate, TState invokeState, Object tag, Int32 timeout, SynchronizationContext callbackSyncContext)
   at System.Web.Mvc.Controller.BeginExecute(RequestContext requestContext, AsyncCallback callback, Object state)
   at System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__4(AsyncCallback asyncCallback, Object asyncState, ProcessRequestState innerState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallBeginDelegate(AsyncCallback callback, Object callbackState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout)
   at System.Web.Mvc.Async.AsyncResultWrapper.Begin[TState](AsyncCallback callback, Object callbackState, BeginInvokeDelegate`1 beginDelegate, EndInvokeVoidDelegate`1 endDelegate, TState invokeState, Object tag, Int32 timeout, SynchronizationContext callbackSyncContext)
   at System.Web.Mvc.MvcHandler.BeginProcessRequest(HttpContextBase httpContext, AsyncCallback callback, Object state)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Update 1

I've been able to find a method to replicate on the test environment:

1. Put a constant load on the app (as few as 20 virtual users)
2. Manually recycle the app pool (repeatedly until failure triggered)

Recycling the app pool without load does not seem to trigger the failure.

More worrying, setting "Disable Overlapped Recycle" to True doesn't stop them from occurring. This, I had assumed would stop any recycling issues as it tears the worker process down completely before starting a new one.

Update 2

Global.asax and associated config doesn’t look (to me) to be too far off the default templates. See what you think?

    public class MyHttpApplication : HttpApplication {

            public MyHttpApplication() {
                    SimpleInjectorConfig.InitializeContainer(); //seems to need to be in ctor to support HttpModule dependencies
            }

            protected void Application_Start() {
                    MvcHandler.DisableMvcResponseHeader = true;

                    AreaRegistration.RegisterAllAreas();

                    ViewEngineConfig.RegisterViewEngines(ViewEngines.Engines);
                    WebApiConfig.Register(GlobalConfiguration.Configuration);
                    FilterConfig.RegisterGlobalFilters(GlobalFilters.Filters);
                    RouteConfig.RegisterRoutes(RouteTable.Routes);
                    BinderConfig.RegisterGlobalBinders(ModelBinders.Binders);
            }

The various XxxConfig classes are pretty much out of the template.

    public static class FilterConfig {
            public static void RegisterGlobalFilters(GlobalFilterCollection filters) {
                    filters.Add(new AuthorizeAttribute());
            }
    }

    public static class SimpleInjectorConfig {
            /// <summary>Initialize the container and register it as MVC Dependency Resolver.</summary>
            public static void InitializeContainer() {
                    var container = new Container();
                    container.Options.AllowResolvingFuncFactories();

                    RegisterServices(container);

                    container.RegisterMvcControllers(Assembly.GetExecutingAssembly());
                    container.RegisterMvcIntegratedFilterProvider();
                    container.RegisterWebApiControllers(GlobalConfiguration.Configuration);

                    //container.Verify(); //see http://bit.ly/YE8OJj for more info

                    DependencyResolver.SetResolver(new SimpleInjectorDependencyResolver(container));
                    GlobalConfiguration.Configuration.DependencyResolver = new SimpleInjectorWebApiDependencyResolver(container);
            }

Update 3

When run with overlapped recycling off, I was able to obtain the following variant stack traces.

[IndexOutOfRangeException: Index was outside the bounds of the array.]
   System.Collections.Generic.Enumerator.MoveNext() +112
   System.Linq.<ConcatIterator>d__71`1.MoveNext() +643
   System.Linq.Buffer`1..ctor(IEnumerable`1 source) +520
   System.Linq.Enumerable.ToArray(IEnumerable`1 source) +103
   System.Web.Mvc.FilterProviderCollection.GetFilters(ControllerContext controllerContext, ActionDescriptor actionDescriptor) +89
   System.Web.Mvc.ControllerActionInvoker.GetFilters(ControllerContext controllerContext, ActionDescriptor actionDescriptor) +38
   System.Web.Mvc.Async.AsyncControllerActionInvoker.BeginInvokeAction(ControllerContext controllerContext, String actionName, AsyncCallback callback, Object state) +333
   System.Web.Mvc.Controller.<BeginExecuteCore>b__1c(AsyncCallback asyncCallback, Object asyncState, ExecuteCoreState innerState) +45
   System.Web.Mvc.Async.WrappedAsyncVoid`1.CallBeginDelegate(AsyncCallback callback, Object callbackState) +111
   System.Web.Mvc.Async.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout) +150
   System.Web.Mvc.Async.AsyncResultWrapper.Begin(AsyncCallback callback, Object callbackState, BeginInvokeDelegate`1 beginDelegate, EndInvokeVoidDelegate`1 endDelegate, TState invokeState, Object tag, Int32 timeout, SynchronizationContext callbackSyncContext) +203
   System.Web.Mvc.Controller.BeginExecuteCore(AsyncCallback callback, Object state) +879
   System.Web.Mvc.Async.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout) +150
   System.Web.Mvc.Async.AsyncResultWrapper.Begin(AsyncCallback callback, Object callbackState, BeginInvokeDelegate`1 beginDelegate, EndInvokeVoidDelegate`1 endDelegate, TState invokeState, Object tag, Int32 timeout, SynchronizationContext callbackSyncContext) +154
   System.Web.Mvc.Controller.BeginExecute(RequestContext requestContext, AsyncCallback callback, Object state) +527
   System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__4(AsyncCallback asyncCallback, Object asyncState, ProcessRequestState innerState) +108
   System.Web.Mvc.Async.WrappedAsyncVoid`1.CallBeginDelegate(AsyncCallback callback, Object callbackState) +111
   System.Web.Mvc.Async.WrappedAsyncResultBase`1.Begin(AsyncCallback callback, Object state, Int32 timeout) +150
   System.Web.Mvc.Async.AsyncResultWrapper.Begin(AsyncCallback callback, Object callbackState, BeginInvokeDelegate`1 beginDelegate, EndInvokeVoidDelegate`1 endDelegate, TState invokeState, Object tag, Int32 timeout, SynchronizationContext callbackSyncContext) +203
   System.Web.Mvc.MvcHandler.BeginProcessRequest(HttpContextBase httpContext, AsyncCallback callback, Object state) +665
   System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +12638163
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +288

and

[IndexOutOfRangeException: Index was outside the bounds of the array.]
   System.Collections.Generic.Enumerator.MoveNext() +112
   System.Linq.<OfTypeIterator>d__aa`1.MoveNext() +344
   System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) +536
   System.Linq.Enumerable.ToList(IEnumerable`1 source) +80
   SimpleInjector.SimpleInjectorMvcExtensions.RegisterMvcIntegratedFilterProvider(Container container) +271
   WebProject.SimpleInjectorConfig.InitializeContainer() in c:\...\App_Start\SimpleInjectorConfig.cs:35
   WebProject.MyHttpApplication..ctor() in c:\...\Global.asax.cs:14
   ASP.global_asax..ctor() in c:\...\App_global.asax.3szzcx02.0.cs:0

[TargetInvocationException: Exception has been thrown by the target of an invocation.]
   System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck) +0
   System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark) +159
   System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark) +256
   System.Activator.CreateInstance(Type type, Boolean nonPublic) +127
   System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark) +14259433
   System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes) +200
   System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture) +28
   System.Web.HttpRuntime.CreateNonPublicInstance(Type type, Object[] args) +83
   System.Web.HttpApplicationFactory.GetNormalApplicationInstance(HttpContext context) +246
   System.Web.HttpApplicationFactory.GetApplicationInstance(HttpContext context) +178
   System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +333

Answer 1

The following line is causing you trouble:

public MyHttpApplication() {
    SimpleInjectorConfig.InitializeContainer(); //seems to need to be in c...    
}

The MyHttpApplication can be called multiple times in that causes multiple containers to be created that each try to register itself as filter provider using RegisterMvcIntegratedFilterProvider. You should make sure that only one container instance is created for that app domain so you only have to call RegisterMvcIntegratedFilterProvider once.