ASP.NET MVC Recompilation limit of 15 reached HostingEnvironment initiated shutdown HostingEnvironment caused shutdown

Question

At some point, shortly after a code push, we saw numerous restarts occurring in our web application with no logging indicating an issue whatsoever. So I found this article: http://weblogs.asp.net/scottgu/433194 and we added Application_End logging, which immediately revealed this:

_shutDownMessage=Recompilation limit of 15 reached HostingEnvironment initiated shutdown HostingEnvironment caused shutdown _shutDownStack= at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) at System.Environment.get_StackTrace() at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal() at System.Web.Hosting.HostingEnvironment.InitiateShutdownWithoutDemand() at System.Web.HttpRuntime.ShutdownAppDomain(String stackTrace) at System.Web.Compilation.DiskBuildResultCache.ShutdownCallBack(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch()

Googling this error reveals surprisingly little, so we updated our web.config

 <compilation debug="false" numRecompilesBeforeAppRestart="100">

and viola! Everything back to normal. We reviewed our changes thoroughly but didn't find anything that we felt could've been the culprit.

Has anyone else encountered this or similar, or has knowledge / suspicions of what in the world could've caused this? Any feedback would be superb!

Answer 1

So the application is actually getting recompiled and since the default limit is 15 after 15 recompilation the app domain/application pool recycles.

Usually you will see an event in event viewer with event id - 1305. Open IIS Manager => Application Pools =>Right Click the Application pool and go to advanced settings => Scroll down to Generate Recycle Event Log Entry and change everything to true. Also you may need to enable health monitoring to see the details in event viewer.

When you increased it to 100 the limit is not reached and hence application pool would have recyled during the regular recycle schedule (default every 29 hours)

A list of what causes whole website recompile:

1. By default, when any change is made to a top-level file in a Web site, the whole site is recompiled. Top-level files include the global.asax file and all files in the bin/ and App_Code/ folders. Additional details here - https://blogs.msdn.microsoft.com/tmarq/2007/11/01/asp-net-file-change-notifications-exactly-which-files-and-directories-are-monitored/
2. modifying web.config

3. a configuration include file change, if the SectionInformation.RestartOnExternalChanges property is true

   <section name="MyAppSettings" type="System.Configuration.AppSettingsSection, System.Configuration, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" restartOnExternalChanges="true" requirePermission="false" />

Notes:

If you want to be able to change top-level files without causing the whole site to be recompiled, you can set the optimizeCompilations attribute of the compilation element in the Web.config file to true

References:

Understanding ASP.Net dynamic compilations

One of the other common cause for recompilation is due to a file being written to the source code folder i.e. something like writing a log file with in the source code folder or anti virus trying to scan web app folder and probably writing something into it (you can exclude web app folder from anti virus scan and see if it helps).

However to exactly find out what is causing recompilation you need to capture a ETW trace and see. A detailed explanation is given here on how to do that - https://blogs.msdn.microsoft.com/tess/2008/11/06/troubleshooting-appdomain-restarts-and-other-issues-with-etw-tracing/

Also there is a known issue around this is mentioned here - http://support.microsoft.com/kb/319947

Relevant text from the link

However, this problem occurs when you load many new .aspx or .ascx files to the server (for example, 61 files). The server unloads the application when the first 15 files are recompiled and every time another 15 files are recompiled until the server reaches 61. This results in four application restarts even though only one is required.

It talks about memory implications so make sure you do have the periodic application pool recycling enabled.

Hope this helps