Performing async operations that are guaranteed to execute within an IIS web site, even if an app pool is recycled

Question

Certain actions that my web site's users carry out result in the sending of emails. The code that sends emails can block for a while, so I want to do this off their HTTP request handler's thread.

Currently I'm using something like:

ThreadPool.QueueUserWorkItem(o => {
    try
    {
        email.Send();
    }
    catch (Exception ex)
    {
        _log.Error("Error sending email", ex);
    }
});

For the most part, this works. However the web site runs in a hosted environment where the app pool can be recycled.

Every once in a while I don't receive an email that should have been sent, and I suspect that this work item on the threadpool's queue is being dropped during application pool recycling.

How can I perform an ansync operation like this and guarantee that it will complete in such a case?

Answer 1

If your application runs in integrated mode, you can register your mail dispatcher service within the host environment. The host will notify your service before a recycling is done. The host will call your implementation of IRegisteredObject.Stop exactly 2 times. On the first call the host gives you the opportunity to finish the job. If the timeout is reached and your service has not removed itself from the host, then another call is made​, but this time only to notify that the recycling will be made with or without the consent of the service.

This is an example (not tested) of how you can implement the Stop() method:

public class MailDispatchService : IRegisteredObject
{
    private AutoResetEvent _processQueueEvt = new AutoResetEvent();
    private ConcurrentQueue<MailMessage> _queue = new ConcurrentQueue<MailMessage>();
    private Thread _dispatcherThread;
    private volatile bool _enabled = true;

    #region Implementation of IRegisteredObject

    public void Stop(bool immediate)
    {
        if (_dispatcherThread != null && _dispatcherThread.IsAlive)
        {
            // it's not an immediate stop, we can wait for the queue to empty
            if (!immediate)
            {
                // stop accepting new items in the send queue...
                _enabled = false;
                // awake dispatcher thread, so it can quit if the queue is empty
                _processQueueEvt.Set();
                // and wait for a while but not forever.
                _dispatcherThread.Join(TimeSpan.FromSeconds(30));
            }
            else
            {
                // host env will recycle now, nothing to do...
                _dispatcherThread.Abort();
            }
        }
        // remove the service from host
        HostingEnvironment.UnregisterObject(this);
    }

    #endregion

    public void Start()
    {
        _dispatcherThread = new Thread(ProcessQueue);
        _dispatcherThread.Start();
    }

    private void ProcessQueue()
    {
        while (_enabled)
        {
            _processQueueEvt.WaitOne();
            MailMessage message;
            while (_queue.TryDequeue(out message)) { /* send mail ...*/}
        }
    }

    public void DispatchEmail(MailMessage message)
    {
        if (!_enabled) throw new Exception("....");
        _queue.Enqueue(message);
        _processQueueEvt.Set();
    }
}

Start the service and register it on the host.

var mailService = new MailDispatchService();
System.Web.Hosting.HostingEnvironment.RegisterObject(mailService);
mailService.Start();

var message = new MailMessage();
mailService.DispatchEmail(message);   

Answer 2

Best option here would be to use a persistent message queue or service bus. Your app writes email request to queue, queue handler (which could be your web app) reads queue and processes message. If things die, the persistence angle kicks in -- the messages hang around until they can be processed.