Native image for System.Data.Entity crashing IIS

Question

I have an MVC 3 website using entity framework and it has just recently started crashing on a specific action. I wish I had more information, but all I can pull from the Windows event viewer is this:

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7a5f8
Faulting module name: System.Data.Entity.ni.dll, version: 4.0.30319.1, time stamp: 0x4ba1e2fd
Exception code: 0xc00000fd
Fault offset: 0x003aac6a
Faulting process id: 0x2f0
Faulting application start time: 0x01ce540a70477360
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Windows\assembly\NativeImages_v4.0.30319_32\System.Data.Entity\dc0c74bc42bbaeffcb7158c7ed0f1653\System.Data.Entity.ni.dll
Report Id: 3659a820-bffe-11e2-8207-404094d3cf82

If I stop IIS, delete the native image, and then restart IIS everything is fine. However, after a day or two the problem will come right back.

Does anyone have ideas on this? It seems as if something is happening in an image created automatically by ngen.exe, but I have no idea where to start debugging. Thanks!

Update

I was finally able to catch a complete memory dump of the fault. It's a stack overflow problem at the core. Here is the stack trace, with the repeating bits...

System.Data.Query.InternalTrees.BasicOpVisitorOfT`1[[System.__Canon, mscorlib]].VisitNode(System.Data.Query.InternalTrees.Node)     086a9990     0609755c     0f16e1a0     59c4a149    
System.Data.Query.InternalTrees.NodeInfoVisitor.RecomputeNodeInfo(System.Data.Query.InternalTrees.Node)     01f8f258     0609755c     06412d6c     0f16e1ec    
System.Data.Query.InternalTrees.Node.InitializeNodeInfo(System.Data.Query.InternalTrees.Command)     086a97f0     59c64fb9     0f16e1b8     5a1d255e    
System.Data.Query.InternalTrees.Node.GetExtendedNodeInfo(System.Data.Query.InternalTrees.Command)     06412da4     0f16e204     59c64ef1     06412da4    
System.Data.Query.InternalTrees.BasicOpVisitorOfT`1[[System.__Canon, mscorlib]].Visit(System.Data.Query.InternalTrees.OuterApplyOp, System.Data.Query.InternalTrees.Node)     06412da4     0f16e220     59c12799     5996fb84    
System.Data.Query.InternalTrees.OuterApplyOp.Accept[[System.__Canon, mscorlib]](System.Data.Query.InternalTrees.BasicOpVisitorOfT`1<System.__Canon>, System.Data.Query.InternalTrees.Node)     5996fb84     06412da4     01f8f098     06412da4    
System.Data.Query.InternalTrees.BasicOpVisitorOfT`1[[System.__Canon, mscorlib]].VisitNode(System.Data.Query.InternalTrees.Node)     086a97f0     06412da4     0f16e244     59c4a149    
System.Data.Query.InternalTrees.NodeInfoVisitor.RecomputeNodeInfo(System.Data.Query.InternalTrees.Node)     01f8f258     06412da4     086a9650     0f16e2b4    
System.Data.Query.InternalTrees.Node.InitializeNodeInfo(System.Data.Query.InternalTrees.Command)     06097914     59c4a751     0f16e26c     00000000    
System.Data.Query.InternalTrees.Node.GetExtendedNodeInfo(System.Data.Query.InternalTrees.Command)     0609794c     0f16e2dc     59c12799     599713cc    
System.Data.Query.InternalTrees.ProjectOp.Accept[[System.__Canon, mscorlib]](System.Data.Query.InternalTrees.BasicOpVisitorOfT`1<System.__Canon>, System.Data.Query.InternalTrees.Node)     599713cc     0609794c     01f8f098     0609794c    
System.Data.Query.InternalTrees.BasicOpVisitorOfT`1[[System.__Canon, mscorlib]].VisitNode(System.Data.Query.InternalTrees.Node)     086a9650     0609794c     0f16e300     59c4a149    
System.Data.Query.InternalTrees.NodeInfoVisitor.RecomputeNodeInfo(System.Data.Query.InternalTrees.Node)     086a94b0     0609794c     0f16e334     0f16e354    
System.Data.Query.InternalTrees.Node.InitializeNodeInfo(System.Data.Query.InternalTrees.Command)     086a94b0     03be2b5c     01f8f258     06097c50    
System.Data.Query.InternalTrees.NodeInfoVisitor.Visit(System.Data.Query.InternalTrees.PhysicalProjectOp, System.Data.Query.InternalTrees.Node)     06097c88     0f16e37c     59c12799     59971884    
System.Data.Query.InternalTrees.PhysicalProjectOp.Accept[[System.__Canon, mscorlib]](System.Data.Query.InternalTrees.BasicOpVisitorOfT`1<System.__Canon>, System.Data.Query.InternalTrees.Node)     59971884     06097c88     01f8f098     06097c88    
System.Data.Query.InternalTrees.BasicOpVisitorOfT`1[[System.__Canon, mscorlib]].VisitNode(System.Data.Query.InternalTrees.Node)     086a94b0     06097c88     0f16e3a0     59c4a149    
System.Data.Query.InternalTrees.NodeInfoVisitor.RecomputeNodeInfo(System.Data.Query.InternalTrees.Node)     06097c88     06097c88     086a9460     0f16e3d4    
System.Data.Query.InternalTrees.Node.InitializeNodeInfo(System.Data.Query.InternalTrees.Command)     086a9494     59c49e5c     0f16e3c8     59c0ffb8    
System.Data.Query.PlanCompiler.TransformationRulesContext.GetHashCode(System.Data.Query.InternalTrees.Node)     00000000     00000000     06097c88     01deb0f0    
System.Data.Query.PlanCompiler.TransformationRules.Process(System.Data.Query.PlanCompiler.PlanCompiler, System.Collections.ObjectModel.ReadOnlyCollection`1<System.Collections.ObjectModel.ReadOnlyCollection`1<System.Data.Query.InternalTrees.Rule>>, Boolean ByRef)     0f16e40c     00000000     00000000     06b605bc    
System.Data.Query.PlanCompiler.TransformationRules.Process(System.Data.Query.PlanCompiler.PlanCompiler, System.Data.Query.PlanCompiler.TransformationRulesGroup)     05d6bd90     06a8ac64     06b605bc     0f16e4f4    
System.Data.Query.PlanCompiler.PlanCompiler.Compile(System.Collections.Generic.List`1<System.Data.Query.PlanCompiler.ProviderCommandInfo> ByRef, System.Data.Query.InternalTrees.ColumnMap ByRef, Int32 ByRef, System.Data.Common.Utils.Set`1<System.Data.Metadata.Edm.EntitySet> ByRef)     01f8f04c     0f16e4ec     0f16e4f0     00000000    
System.Data.EntityClient.EntityCommandDefinition..ctor(System.Data.Common.DbProviderFactory, System.Data.Common.CommandTrees.DbCommandTree)     01f8efe0     01de3f40     01c645c4     0213e214    
System.Data.EntityClient.EntityProviderServices.CreateCommandDefinition(System.Data.Common.DbProviderFactory, System.Data.Common.CommandTrees.DbCommandTree)     0213e214     01de3f40     01c645c4     0213e214    
System.Data.EntityClient.EntityProviderServices.CreateDbCommandDefinition(System.Data.Common.DbProviderManifest, System.Data.Common.CommandTrees.DbCommandTree)     0213e214     0208a8f0     01f47934     0213e214    
System.Data.Common.DbProviderServices.CreateCommandDefinition(System.Data.Common.CommandTrees.DbCommandTree)     00000000     00000000     00000000     00000000    
System.Data.Objects.Internal.ObjectQueryExecutionPlan.Prepare(System.Data.Objects.ObjectContext, System.Data.Common.CommandTrees.DbQueryCommandTree, System.Type, System.Data.Objects.MergeOption, System.Data.Objects.Span, System.Collections.ObjectModel.ReadOnlyCollection`1<System.Collections.Generic.KeyValuePair`2<System.Data.Objects.ObjectParameter,System.Data.Objects.ELinq.QueryParameterExpression>>)     00000000     00000000     00000003     01f5b740    
System.Data.Objects.ELinq.ELinqQueryState.GetExecutionPlan(System.Nullable`1<System.Data.Objects.MergeOption>)     00000000     00000000     01f5c228     00000000    
System.Data.Objects.ObjectQuery`1[[System.__Canon, mscorlib]].GetResults(System.Nullable`1<System.Data.Objects.MergeOption>)     00000000     00000000     0f16e670     0208a8cc    
System.Data.Objects.ObjectQuery`1[[System.__Canon, mscorlib]].System.Collections.Generic.IEnumerable<T>.GetEnumerator()     00000000     00000000     0f16e704     00000000    
System.Linq.Buffer`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>)     0f4cb654     00000000     00000000     00000000    
System.Linq.Enumerable.ToArray[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)     02089768     020883d4     020881fc     02088104    
OptimizerCore.Models.agField.GetFieldPreloadMidpointInformation(System.Collections.Generic.IEnumerable`1<Int32>)     05e2e50c     05e2e50c     5d664381     05db9a64    
System.Web.Mvc.ActionMethodDispatcher.Execute(System.Web.Mvc.ControllerBase, System.Object[])     05db9a64     05e22d60     05daee58     01d68350    
System.Web.Mvc.ReflectedActionDescriptor.Execute(System.Web.Mvc.ControllerContext, System.Collections.Generic.IDictionary`2<System.String,System.Object>)     05daf334     05e23b4c     05e23b0c     05db9a3c    
System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(System.Web.Mvc.ControllerContext, System.Web.Mvc.ActionDescriptor, System.Collections.Generic.IDictionary`2<System.String,System.Object>)     05daf334     05e22ca0     05e23bcc     00000000    
System.Web.Mvc.ControllerActionInvoker+<>c__DisplayClass15.<InvokeActionMethodWithFilters>b__12()     0f16e90c     05daf394     05daf1b0     00000000    
System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(System.Web.Mvc.IActionFilter, System.Web.Mvc.ActionExecutingContext, System.Func`1<System.Web.Mvc.ActionExecutedContext>)     05daf3b8     05e23c00     5d66622e     61954b4e    
System.Web.Mvc.ControllerActionInvoker+<>c__DisplayClass15+<>c__DisplayClass17.<InvokeActionMethodWithFilters>b__14()     05daf3ec     05e23b0c     5d666580     05e22ca0    
System.Web.Mvc.ControllerActionInvoker+<>c__DisplayClass15+<>c__DisplayClass17.<InvokeActionMethodWithFilters>b__14()     05daf334     05daee58     05daf0b4     05daef68    
System.Web.Mvc.ControllerActionInvoker.InvokeAction(System.Web.Mvc.ControllerContext, System.String)     05bf96e4     05bf961c     05c0974c     00000000    
System.Web.Mvc.Controller.ExecuteCore()     01d6842c     00000000     00000000     00000000    
System.Web.Mvc.ControllerBase.Execute(System.Web.Routing.RequestContext)     0f16eb08     5d672ca6     05bfdc94     0f16eb2c    
System.Web.Mvc.ControllerBase.System.Web.Mvc.IController.Execute(System.Web.Routing.RequestContext)     05bfdc94     0f16eb2c     5d66ad06     00000000    
System.Web.Mvc.MvcHandler+<>c__DisplayClass6+<>c__DisplayClassb.<BeginProcessRequest>b__5()     00000000     01dcf71c     5d68febd     5d68fcdf    
System.Web.Mvc.Async.AsyncResultWrapper+<>c__DisplayClass1.<MakeVoidDelegate>b__0()     5d5b8c28     01dcf7ac     0f16eb4c     5d65aaf8    
System.Web.Mvc.MvcHandler+<>c__DisplayClasse.<EndProcessRequest>b__d()     5d65ab27     01dcf7a0     0f16eb5c     5d67244d    
System.Web.Mvc.SecurityUtil.<GetCallInAppTrustThunk>b__0(System.Action)     01dcf71c     05bfbb70     0f16eb64     5d67233a    
System.Web.Mvc.MvcHandler.EndProcessRequest(System.IAsyncResult)     0f16eb90     5e0ef242     05be7470     00000000    
System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(System.IAsyncResult)     05be7470     00000000     00000000     00000000    
System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()     0f16ec5c     61d1965e     61d19673     00000000    
System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)     0f16ec24     05d11148     00000000     05bebd0c    
System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)     00000000     00000000     0f16eca0     0f16ecb0    
System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)     05bc3134     05be7294     05bc2e34     00000080    
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)     05be7470     9572145a     61cf5d80     0f16f218    
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)     00000000     0000000c     00000000     00000000    
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)     00c11198     01872c94     0000000c     00000000    
webengine4!W3_MGD_HANDLER::ProcessNotification+5b     01871ba8     00000100     00000000     0f16ee1c    
webengine4!W3_MGD_HANDLER::DoWork+250     00000080     00000000     00efb22c     01871c00    
webengine4!RequestDoWork+2da     00000080     00000000     00efb22c     01871ba8    
webengine4!CMgdEngHttpModule::OnExecuteRequestHandler+1a     01871ba8     01871c00     01871bac     0f16eec8    
iiscore!NOTIFICATION_CONTEXT::RequestDoWork+128     00efb22c     01871ba8     00000000     01871c00    
iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+305     00000000     00000000     00000000     01871ba8    
iiscore!NOTIFICATION_CONTEXT::CallModules+28     00000000     00000000     00000000     01871ba8    
iiscore!W3_CONTEXT::DoStateRequestExecuteHandler+36     00000000     00000000     00000000     00000000    
iiscore!W3_CONTEXT::DoWork+d7     00000000     00000000     00000001     01871ba8    
iiscore!W3_MAIN_CONTEXT::ContinueNotificationLoop+1f     00000000     00000000     768c1484     0f16f1e0    
iiscore!W3_MAIN_CONTEXT::ProcessIndicateCompletion+1f     00000000     01872c94     0f16f1f4     71b4d641    
iiscore!W3_CONTEXT::IndicateCompletion+75     00000000     0f16f2a8     0f16f218     0f16f204    
webengine4!W3_MGD_HANDLER::IndicateCompletion+3d     00000000     0e8ecc18     0f16f254     5d8d8bbd    
webengine4!MgdIndicateCompletion+24     01872c94     0f16f2a8     9572145a     61cf5d80    
DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)     9572145a     61cf5d80     0f16f3ec     5d7467e4    
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)     00000004     00000003     00000000     00000000    
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)     0e8ecc18     0f16f3a0     61fbf077     0f16f510    
clr!UM2MThunk_WrapperHelper+10     0f16f510     00000010     00671fa3     00672038    
clr!UM2MThunk_Wrapper+8c     0f16f4a8     9a0857ca     0e8ecc18     00000002    
clr!Thread::DoADCallBack+f0     00000002     61fbefeb     0f16f4a8     00000001    
clr!UM2MDoADCallBack+c0     01924570     ffffffff     0f16f510     00000010    
0x00672020     00000000     01872c94     00000003     00000004    
webengine4!W3_MGD_HANDLER::ProcessNotification+5b     01872c94     71b508c8     0f16f58c     61e75eaf    
webengine4!ProcessNotificationCallback+36     01872c94     9a0856c6     0000ffff     00010002    
clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+195     0f16f5ef     0f16f5ee     9a0856b6     00000000    
clr!ThreadpoolMgr::NewWorkerThreadStart+20b     00000000     9a08552e     00000000     61e76a99    
clr!ThreadpoolMgr::WorkerThreadStart+3d1     00000000     0000027f     00000000     00000000    
clr!Thread::intermediateThreadProc+4b     0e8b0b60     0f16f854     778e9f42     0e8b0b60    
kernel32!BaseThreadInitThunk+e     0e8b0b60     78e16e6f     00000000     00000000    
ntdll!__RtlUserThreadStart+70     61da59c0     0e8b0b60     ffffffff     779775ad    
ntdll!_RtlUserThreadStart+1b 

This bit in particular seems to go recursive and eventually hard fault:

System.Data.Query.InternalTrees.Node.GetExtendedNodeInfo(System.Data.Query.InternalTrees.Command)     0609794c     0f16e2dc     59c12799     599713cc    
System.Data.Query.InternalTrees.ProjectOp.Accept[[System.__Canon, mscorlib]](System.Data.Query.InternalTrees.BasicOpVisitorOfT`1<System.__Canon>, System.Data.Query.InternalTrees.Node)     599713cc     0609794c     01f8f098     0609794c    
System.Data.Query.InternalTrees.BasicOpVisitorOfT`1[[System.__Canon, mscorlib]].VisitNode(System.Data.Query.InternalTrees.Node)     086a9650     0609794c     0f16e300     59c4a149    
System.Data.Query.InternalTrees.NodeInfoVisitor.RecomputeNodeInfo(System.Data.Query.InternalTrees.Node)     086a94b0     0609794c     0f16e334     0f16e354    
System.Data.Query.InternalTrees.Node.InitializeNodeInfo(System.Data.Query.InternalTrees.Command) 

I believe this is the chunk of code that causes the problem: (where fieldIDs is an int array containing about 1,600 items):

var query = db.BaseDB.agZoneCoordinates
                        .Where(x => fieldIDs.Contains(x.agZone.FieldID))
                        .Select(x => new {
                            x.agZone.FieldID,
                            x.Latitude,
                            x.Longitude
                        })
                        .AsNoTracking()
                        .ToArray();

I've tried reducing the query size so that it works in chunks of 200 or so, but it's still crashing. Are there any Entity Framework developers out there that have seen this happen? What's making EF go recursive?

Answer 1

We recently ran into a stack overflow very much like yours where we were doing something very similar to your scenario (a large list of Ids) and EF usage except we were also calling Distinct on the list of Ids we were passing. So, building on your example, our call looked something like:

var query = db.BaseDB.agZoneCoordinates
               .Where(x => fieldIDs.Distinct().Contains(x.agZone.FieldID))
               .Select(x => new {
                   x.agZone.FieldID,
                   x.Latitude,
                   x.Longitude
                   })
               .AsNoTracking()
               .ToArray();

Since the lambda is for an IQueryable, this has the effect of creating a huge expression tree that EF has to deal with and, apparently, it doesn't handle it well. We solved our problem and actually improved performance for large lists by doing the Distinct call outside of the lambda. FWIW, I tried it w/ a large list (100K) w/o the Distinct call and that also worked so our issue was definitely w/ having the Distinct call as part of the lambda rather than the sheer size of the list.