Graceful Shutdown of Azure WebJobs

Question

I'm planning to use a continuous Azure WebJob to send emails and push notifications. I understand that WebJobs will be started and stopped from time to time for various reasons. That's fine, but I would like to have the opportunity to "clean up" prior to the job getting shut down.

That way my WebJob can update the status of database records or delete queue items that have already been processed in each batch so duplicate messages won't get sent the next time the job runs.

As a shot on the dark, I tried adding the following code to my C# console application:

Console.CancelKeyPress += (object sender, ConsoleCancelEventArgs e) =>
{
  e.Cancel = true;
  program.keepRunning = false;
};

And then I used the keepRunning bool to control the main while loop, and put a Console.Writeline("Exited Gracefully") outside of the while loop. But that didn't seem to help. When I tell the job to stop (using the stop button from the Webjobs tab of my Azure Website), the job disappears from the list and it says "Could not stop job: 'JobName'." in the Azure portal (at the bottom of the page). I don't see the "Exited Gracefully" text in the WebJob's log. So I took that code out since it wasn't helping.

So, I'm looking for a good way for my WebJob to be notified that its seconds are numbered and that it needs to get its affairs in order.

Answer 1

I believe things changed soon after Amit answered last, according to his own blog post in fact here: WebJobs Graceful Shutdown

Also see this video a little after the 6:00+ mark for some discussion on this.

From Amit's blog:

The way Azure notifies the process it's about to be stopped is by placing (creating) a file at a path that is passed as an environment variable called WEBJOBS_SHUTDOWN_FILE.

Any WebJob that wants to listen on the shutdown notification will actually have to check for the presence of the file (using simple File.Exists function or using a FileSystemWatcher in whatever script language you use), when it shows up the WebJob will need to start cleaning up and break it's current loop where preferably it'll exit properly and Azure will continue the shutdown (of the site) process.

Well, that doesn't sound like much fun having to handle. While Amit and others have posted some code to handle this (see that post), I found it still more unwieldy than I would like (I prefer ugly details to be taken care of in code once, and then promptly relied on and forgotten). I hope the following is a better improvement. I really wanted a single line notification of shutdown setup, and that's what we have now with the following. I just tested this solution, shut down my job and it fired correctly.

All the work is placed in a separate file / type, which I named WebJobShutdownNotifier. First, the usage: just instantiate this type in your Main method and pass a void function (or lamda) having the shutdown work. That's it! It will fire your Shutdown method, not much else to say about it. I suggest to the WebJobs team they incorporate this or something like it directly within the JobHost. Just provide an event to subscribe to.

Example usage:

    public static void Main() // your Main method...
    {
        // nice! a single line to handle the shutdown notification, firing your IsShuttingDown method
        var shutdownNotifier = new WebJobShutdownNotifier(IsShuttingDown);

        var host1 = new JobHost();
        host1.RunAndBlock();
    }

    public static void IsShuttingDown()
    {
        Console.WriteLine("Were shutin' down the webjob hatches baby! - {0}", DateTime.UtcNow);
        // do something else here if needed...
    }

// --- WebJobShutdownNotifier.cs ---

using System;
using System.IO;

namespace Microsoft.Azure.WebJobs.Helper
{
    /// <summary>
    /// Base info and code adapted and expanded from Amit Apple:
    /// http://blog.amitapple.com/post/2014/05/webjobs-graceful-shutdown/.
    /// To change the wait on shutdown time from the default of 5 seconds:
    /// "create a file called settings.job with the following content: { "stopping_wait_time": 60 }""
    /// (Nicholas Petersen)
    /// </summary>
    public class WebJobShutdownNotifier
    {
        public bool IsRunning { get; private set; }

        public string ShutdownFilePath { get; private set; }

        public bool FileEnvironmentVariableExisted { get; private set; }

        /// <summary>
        /// Set this as an action allowing you to be notified when it fires that 
        /// shutdown has been triggered (/detected).
        /// </summary>
        public Action IsShuttingDownNotifier { get; set; }

        /// <summary>
        /// Constructor.
        /// </summary>
        /// <param name="isShuttingDownNotifier">
        /// Set this as an action allowing you to be notified when it fires that 
        /// shutdown has been triggered (/detected).
        /// </param>
        public WebJobShutdownNotifier(Action isShuttingDownNotifier = null, bool exceptionIfNoFileEnvironmentVariable = false)
        {
            IsRunning = true;
            IsShuttingDownNotifier = isShuttingDownNotifier;

            // Get the shutdown file path from the environment
            ShutdownFilePath = Environment.GetEnvironmentVariable("WEBJOBS_SHUTDOWN_FILE");

            FileEnvironmentVariableExisted = !string.IsNullOrEmpty(ShutdownFilePath);

            if (!FileEnvironmentVariableExisted) {
                if (exceptionIfNoFileEnvironmentVariable)
                    throw new Exception("WEBJOBS_SHUTDOWN_FILE Environment variable returned null or empty.");
            }
            else {
                // Setup a file system watcher on that file's directory to know when the file is created
                var fileSystemWatcher = new FileSystemWatcher(Path.GetDirectoryName(ShutdownFilePath));
                fileSystemWatcher.Created += OnChanged;
                fileSystemWatcher.Changed += OnChanged;
                fileSystemWatcher.NotifyFilter = NotifyFilters.CreationTime | NotifyFilters.FileName | NotifyFilters.LastWrite;
                fileSystemWatcher.IncludeSubdirectories = false;
                fileSystemWatcher.EnableRaisingEvents = true;
            }
        }

        private void OnChanged(object sender, FileSystemEventArgs e)
        {
            if (IsRunning) { // this was hitting more than once in the short shut down time, do not want to fire IsShuttingDownNotifier more than once...
                if (e.FullPath.IndexOf(Path.GetFileName(ShutdownFilePath), StringComparison.OrdinalIgnoreCase) >= 0) {
                    // Found the file mark, this WebJob has finished
                    IsRunning = false;
                    if (IsShuttingDownNotifier != null)
                        IsShuttingDownNotifier();
                }
            }
        }

    }
}