How to properly use IRegisteredObject to block app domain shutdown / recycle for web app?

Question

I have a .NET MVC web app which requires time to be properly shutdown and so whenever the IIS app domain is recycled (i.e. a new instance is spun up and receives all new requests while the old instance shuts down waiting for outstanding requests to complete) I need to block this app shutdown until my app's current async background work (containing no outstanding requests) has completed. IRegisteredObject (see http://blog.stephencleary.com/2014/06/fire-and-forget-on-asp-net.html) offers this blocking ability, though, my processes always seem to die at times inconsistent with my blockage time and IIS settings.

I saw this post (IRegisteredObject not working as expected) which explained the importance of the IIS Shutdown Time Limit but, while IRegisteredObject seems to block for a period of time, I cannot get the recycle to block for the desired time of 2 hours (nor can I generally get results which make sense based off various settings).

Below is a simple implementation of IRegisteredObject with a background thread that I've been using for tests:

public class MyRegisteredObject : IRegisteredObject
{
    public void Register()
    {
        HostingEnvironment.RegisterObject(this);
        Logger.Log("Object has been registered");
    }

    // the IRegisteredObject.Stop(...) function gets called on app domain recycle.
    // first, it calls with immediate:false, indicating to shutdown work, then it
    // calls 30s later with immediate:true, and this call 'should' block recycling
    public void Stop(bool immediate)
    {
        Logger.Log("App domain stop has been called: " 
            + (immediate ? "Immediate" : "Not Immediate")
            + " Reason: " + HostingEnvironment.ShutdownReason);
        if (immediate)
        {
            // block for a super long time
            Thread.Sleep(TimeSpan.FromDays(1));
            Logger.Log("App domain immediate stop finished");
        }
    }

    // async background task to track if our process is still alive
    public async Task RunInBackgroundAsync()
    {
        Logger.Log("Background task started");
        var timeIncrement = TimeSpan.FromSeconds(5);
        var time = TimeSpan.Zero;
        while (time < TimeSpan.FromDays(1))
        {
            await Task.Delay(timeIncrement).ConfigureAwait(false);
            time += timeIncrement;
            Logger.Log("Background task running... (" 
                + time.ToString(@"hh\:mm\:ss") + ")");
        }
        Logger.Log("Background task finished");
    }
}

public static class Logger
{
    private static readonly string OutputFilename = @"C:\TestLogs\OutputLog-" + Guid.NewGuid() + ".log";

    public static void Log(string line)
    {
        lock (typeof(Logger))
        {
            using (var writer = new StreamWriter(OutputFilename, append: true))
            {
                writer.WriteLine(DateTime.Now + " - " + line);
                writer.Close();
            }
        }
    }
}

In app start, I start the IRegisteredObject component:

var recycleBlocker = new MyRegisteredObject();
recycleBlocker.Register();
var backgroundTask = recycleBlocker.RunInBackgroundAsync();

Finally, when testing, I have sparked app domain recycles through 3 separate means:

(1) Web.config file change (yields a HostingEnvironment.ShutdownReason value of ConfigurationChange)

(2) Manual recycle through clicking the app's Application Pool and then Recycle in IIS Manager (yields a HostingEnvironment.ShutdownReason value of HostingEnvironment)

(3) Allowing the app to automatically recycle based off of the IIS setting under Process Model - "Idle Time-out (minutes)" (also yields a HostingEnvironment.ShutdownReason value of HostingEnvironment)

I would not have expected this, but the manner in which the recycle is triggered seems to play a drastic role... below are my findings through tests where I modified the means of recycle and IIS settings (Shutdown limit and Idle time-out).

Findings:

---- Web.config change recycle (ShutdownReason: ConfigurationChange) ----

After the IRegisteredObject(immediate: true) call occurs, I see in my logs that the background task lasts almost exactly the time set for IIS Idle Time-out, while Shutdown Time Limit plays no role whatsoever. Further, with this recycle, assuming I set the Idle time-out high enough, the recycle blocking is always respected. I blocked for a full day in one test by setting the Idle time-out to 0 (i.e. off).

---- IIS Manager manual recycle (ShutdownReason: HostingEnvironment) ----

After the IRegisteredObject(immediate: true) call occurs, the logs show the exact opposite behavior compared to Web.config change. No matter what the Idle Time-out is, the blockage seems unaffected. Conversely, the Shutdown Time Limit dictates how long to block the recycle (up to a point). From 1 second up through 5 minutes, the recycle will be blocked based on this Shutdown Limit. However, if the setting is set higher or turned off, blockage seems to remain at the ceiling of around 5 minutes.

---- Idle time-out automatic recycle (ShutdownReason: HostingEnvironment) ----

Finally something predictable... the automatic recycle does actually get triggered based on the Idle Time-out setting, which then causes a situation similar to the Manual Recycle case: Shutdown Time Limit respected up to about 5 minutes but no longer than that. Presumably this is because the automatic and manual recycles each have the same HostingEnvironment.ShutdownReason: HostingEnvironment.

Ok... I apologize for the length of this! As you can see, the combination of recycle methods and IIS settings simply do not seem to yield expected results. Further, my goal of this all is to be able to block for a max of two hours, which does not seem possible from my tests outside of the web.config recycle case, no matter the settings I choose.... Can someone please shed light onto what exactly is going on under the hood here? What role does ShutdownReason play? What role do these IIS settings play?

Fundamentally, what am I missing here, and how can I use IRegisteredObject to block for longer periods of time caused from automatic recycles??

Answer 1

There are two different concepts in play here-

1. Application domain
2. Application pool

Application domains provide an isolation boundary for security, reliability, and versioning, and for unloading assemblies. Application pools define a set of Web applications that share one or more worker processes. Each application pool could be hosting one or more application domains.

Application pool can be recycled in the following ways

1. Triggering IIS Manager manual recycle
2. Invoking IIS reset from command prompt
3. Setting idle timeout or shutdown time limit on your Application pool

Application domain is recylced when you touch the Web.config, Global.asax or any dll in the bin folder of an application.

IRegisteredObject is able to intercept the process of application domain unloading, however, it does not have any control over the application pool recycle. When application pool recycle is triggered, it kills and restarts the w3wp.exe process and ends up killing all the application domains associated with the application pool.

This should explain why your IRegisteredObject was working as expected when you touched your Web.config but would not perform the expected actions when you recylced the application pool. If your idle-timeout or shut down timeout are less than the time window that your IRegisteredObject keeps the application domain alive, after a Application domain recycle is triggered, the IRegisteredObject will try to keep the application domain alive but when the idle-timeout or the shut-down timeout is reached, the application pool will recycle and application domain will be killed regardless of the IRegisteredObject.

A solution to your problem will be to turn the idle timeout and shut down time limit settings off for the application pool and rely on some alternate method to recycle your application pool. In that case, your application pool will not be recycled automatically and you can rely on your IRegisteredObject to keep the application domain alive.