Exception thrown from task is swallowed, if thrown after 'await'

Question

I'm writing a background service using .NET's HostBuilder. I have a class called MyService that implements BackgroundService ExecuteAsync method, and I encountered some weird behavior there. Inside the method I await a certain task, and any exception thrown after the await is swallowed, but an exception that is thrown before the await terminates the process.

I looked online in all sorts of forums (stack overflow, msdn, medium) but I could not find an explanation for this behavior.

public class MyService : BackgroundService
    {
        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            await Task.Delay(500, stoppingToken);
            throw new Exception("oy vey"); // this exception will be swallowed
        }
    }

public class MyService : BackgroundService
    {
        protected override async Task ExecuteAsync(CancellationToken stoppingToken)
        {
            throw new Exception("oy vey"); // this exception will terminate the process
            await Task.Delay(500, stoppingToken);
        }
    }

I expect both exception to terminate the process.

Answer 1

TL;DR;

Don't let exceptions get out of ExecuteAsync. Handle them, hide them or request an application shutdown explicitly.

Don't wait too long before starting the first asynchronous operation in there either

Explanation

This has little to do with await itself. Exceptions thrown after it will bubble up to the caller. It's the caller that handles them, or not.

ExecuteAsync is a method called by BackgroundService which means any exception raised by the method will be handled by BackgroundService. That code is :

    public virtual Task StartAsync(CancellationToken cancellationToken)
    {
        // Store the task we're executing
        _executingTask = ExecuteAsync(_stoppingCts.Token);

        // If the task is completed then return it, this will bubble cancellation and failure to the caller
        if (_executingTask.IsCompleted)
        {
            return _executingTask;
        }

        // Otherwise it's running
        return Task.CompletedTask;
    }

Nothing awaits the returned task, so nothing is going to throw here. The check for IsCompleted is an optimization that avoids creating the async infrastructure if the task is already complete.

The task won't be checked again until StopAsync is called. That's when any exceptions will be thrown.

    public virtual async Task StopAsync(CancellationToken cancellationToken)
    {
        // Stop called without start
        if (_executingTask == null)
        {
            return;
        }

        try
        {
            // Signal cancellation to the executing method
            _stoppingCts.Cancel();
        }
        finally
        {
            // Wait until the task completes or the stop token triggers
            await Task.WhenAny(_executingTask, Task.Delay(Timeout.Infinite, cancellationToken));
        }

    }

From Service to Host

In turn, the StartAsync method of each service is called by the StartAsync method of the Host implementation. The code reveals what's going on :

    public async Task StartAsync(CancellationToken cancellationToken = default)
    {
        _logger.Starting();

        await _hostLifetime.WaitForStartAsync(cancellationToken);

        cancellationToken.ThrowIfCancellationRequested();
        _hostedServices = Services.GetService<IEnumerable<IHostedService>>();

        foreach (var hostedService in _hostedServices)
        {
            // Fire IHostedService.Start
            await hostedService.StartAsync(cancellationToken).ConfigureAwait(false);
        }

        // Fire IHostApplicationLifetime.Started
        _applicationLifetime?.NotifyStarted();

        _logger.Started();
    }

The interesting part is :

        foreach (var hostedService in _hostedServices)
        {
            // Fire IHostedService.Start
            await hostedService.StartAsync(cancellationToken).ConfigureAwait(false);
        }

All the code up to the first real asynchronous operation runs on the original thread. When the first asynchronous operation is encountered, the original thread is released. Everything after the await will resume once that task completes.

From Host to Main()

The RunAsync() method used in Main() to start the hosted services actually calls the Host's StartAsync but not StopAsync :

    public static async Task RunAsync(this IHost host, CancellationToken token = default)
    {
        try
        {
            await host.StartAsync(token);

            await host.WaitForShutdownAsync(token);
        }
        finally
        {
#if DISPOSE_ASYNC
            if (host is IAsyncDisposable asyncDisposable)
            {
                await asyncDisposable.DisposeAsync();
            }
            else
#endif
            {
                host.Dispose();
            }

        }
    }

This means that any exceptions thrown inside the chain from RunAsync to just before the first async operation will bubble up to the Main() call that starts the hosted services :

await host.RunAsync();

or

await host.RunConsoleAsync();

This means that everything up to the first real await in the list of BackgroundService objects runs on the original thread. Anything thrown there will bring down the application unless handled. Since the IHost.RunAsync() or IHost.StartAsync() are called in Main(), that's where the try/catch blocks should be placed.

This also means that putting slow code before the first real asynchronous operation could delay the entire application.

Everything after that first asynchronous operation will keep running on a threadpool thread. That's why exceptions thrown after that first operation won't bubble up until either the hosted services shut down by calling IHost.StopAsync or any orphaned tasks get GCd

Conclusion

Don't let exceptions escape ExecuteAsync. Catch them and handle them appropriately. The options are :

• Log and "ignore" them. This will live the BackgroundService inoperative until either the user or some other event calls for an application shutdown. Exiting ExecuteAsync doesn't cause the application to exit.
• Retry the operation. That's probably the most common option of a simple service.
• In a queued or timed service, discard the message or event that faulted and move to the next one. That's probably the most resilient option. The faulty message can be inspected, moved to a "dead letter" queue, retried etc.
• Explicitly ask for a shutdown. To do that, add the IHostedApplicationLifetTime interface as a dependency and call StopAsync from the catch block. This will call StopAsync on all other background services too

Documentation

The behaviour of hosted services and BackgroundService is described in Implement background tasks in microservices with IHostedService and the BackgroundService class and Background tasks with hosted services in ASP.NET Core.

The docs don't explain what happens if one of those services throws. They demonstrate specific use scenarios with explicit error handling. The queued background service example discards the message that caused the fault and moves to the next one :

    while (!cancellationToken.IsCancellationRequested)
    {
        var workItem = await TaskQueue.DequeueAsync(cancellationToken);

        try
        {
            await workItem(cancellationToken);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, 
               $"Error occurred executing {nameof(workItem)}.");
        }
    }

Answer 2

You don't have to use BackgroundService. As the name implies, it's useful for work that isn't the primary responsibility of the process and whose errors shouldn't cause it to exit.

You can roll your own IHostedService if this doesn't fit your needs. I've used the below WorkerService, which has some advantages over IApplicationLifetime.StopApplication(). Because async void runs continuations on the thread pool, errors can be handled with AppDomain.CurrentDomain.UnhandledException and will terminate with an error exit code. See XML comments for more details.

using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;

namespace MyWorkerApp.Hosting
{
    /// <summary>
    /// Base class for implementing a continuous <see cref="IHostedService"/>.
    /// </summary>
    /// <remarks>
    /// Differences from <see cref="BackgroundService"/>:
    /// <list type = "bullet">
    /// <item><description><see cref="ExecuteAsync"/> is repeated indefinitely if it completes.</description></item>
    /// <item><description>Unhandled exceptions are observed on the thread pool.</description></item>
    /// <item><description>Stopping timeouts are propagated to the caller.</description></item>
    /// </list>
    /// </remarks>
    public abstract class WorkerService : IHostedService, IDisposable
    {
        private readonly TaskCompletionSource<byte> running = new TaskCompletionSource<byte>();
        private readonly CancellationTokenSource stopping = new CancellationTokenSource();

        /// <inheritdoc/>
        public virtual Task StartAsync(CancellationToken cancellationToken)
        {
            Loop();
            async void Loop()
            {
                if (this.stopping.IsCancellationRequested)
                {
                    return;
                }

                try
                {
                    await this.ExecuteAsync(this.stopping.Token);
                }
                catch (OperationCanceledException) when (this.stopping.IsCancellationRequested)
                {
                    this.running.SetResult(default);
                    return;
                }

                Loop();
            }

            return Task.CompletedTask;
        }

        /// <inheritdoc/>
        public virtual Task StopAsync(CancellationToken cancellationToken)
        {
            this.stopping.Cancel();
            return Task.WhenAny(this.running.Task, Task.Delay(Timeout.Infinite, cancellationToken)).Unwrap();
        }

        /// <inheritdoc/>
        public virtual void Dispose() => this.stopping.Cancel();

        /// <summary>
        /// Override to perform the work of the service.
        /// </summary>
        /// <remarks>
        /// The implementation will be invoked again each time the task completes, until application is stopped (or exception is thrown).
        /// </remarks>
        /// <param name="cancellationToken">A token for cancellation.</param>
        /// <returns>A task representing the asynchronous operation.</returns>
        protected abstract Task ExecuteAsync(CancellationToken cancellationToken);
    }
}