.Net core IHostedService Background task exception will not terminate application

Question

I have a program that needs to terminate when an IHostedService background task encounters a certain scenario. I was hoping to do this by just throwing an exception in the background task that would get kicked up to the main function. I could then trigger the cancellation token to kill other background tasks. My problem is that when I throw the exception, it kills the task and that's all. Everything else keeps running. Is there a way to do this, or a better way to do what I'm trying to do? Is there another way to have a backgrounds task trigger the common CancellationToken?

I included a simplified version of my issue in the code below. If I comment out the await Task.Delay(TimeSpan.FromSeconds(1), stoppingToken); line, the exception does what I want and I can trigger the CancellationToken. When it is in place, the task stops, but the program does not.

NOTE: In my messier code I have more IHostedServices running, which is why I'm trying to trigger cancelSource.Cancel()

public class Program
{
    public static async Task Main(string[] args)
    {
        using (var cancelSource = new CancellationTokenSource())
        {
            try
            {
                await new HostBuilder()
                    .ConfigureServices((hostContext, services) =>
                    {
                        services.AddHostedService<TestService>();
                    })
                    .Build()
                    .RunAsync(cancelSource.Token);
            }
            catch (Exception E)
            {
                cancelSource.Cancel();
            }
        }
    }
}
public class TestService : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {

        while (!stoppingToken.IsCancellationRequested)
        {
            await Task.Delay(TimeSpan.FromSeconds(1), stoppingToken);
            Console.WriteLine("loop 1");
            throw new ApplicationException("OOPS!!");
        }
    }
}

Answer 1

Commenting the only line in the ExecuteAsync method with await operator makes your code run synchronously. If we look at the sources of BackgroundService.StartAsync, we can see that it checks for _executingTask.IsCompleted and it returns task that will contain your exception in case we don't have any await in ExecuteAsync method, otherwise it will return Task.CompletedTask and you won't be able to catch this exception from ExecuteAsync in Main method.

You can manage your services with IApplicationLifetime that can be injected in all your background services. For example, you can catch exception within ExecuteMethod and call ApplicationLifetime.StopApplication.

Example:

static async Task Main(string[] args)
{
    await new HostBuilder()
        .ConfigureServices((hostContext, services) =>
        {
            services.AddHostedService<TestService>();
            services.AddHostedService<TestService2>();
        })
        .Build()
        .RunAsync();

    Console.WriteLine("App stoped");
}

Service 1

public class TestService : BackgroundService
{
    private readonly IApplicationLifetime _applicationLifetime;
    public TestService(IApplicationLifetime applicationLifetime)
    {
        _applicationLifetime = applicationLifetime;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        try
        {
            while (!_applicationLifetime.ApplicationStopping.IsCancellationRequested)
            {
                await Task.Delay(TimeSpan.FromSeconds(1), _applicationLifetime.ApplicationStopping);
                Console.WriteLine("running service 1");
                throw new ApplicationException("OOPS!!");
            }
        }
        catch (ApplicationException)
        {
            _applicationLifetime.StopApplication();
        }
    }
}

Service 2

public class TestService2 : BackgroundService
{
    private readonly IApplicationLifetime _applicationLifetime;
    public TestService2(IApplicationLifetime applicationLifetime)
    {
        _applicationLifetime = applicationLifetime;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        try
        {
            while (!_applicationLifetime.ApplicationStopping.IsCancellationRequested)
            {
                await Task.Delay(100, _applicationLifetime.ApplicationStopping);
                Console.WriteLine("running service 2");
            }
        }
        catch (ApplicationException)
        {
            _applicationLifetime.StopApplication();
        }
    }
}

Output:

running service 2
running service 2
running service 1
App stoped