Graceful shutdown of IHostedService

Question

I am trying to develop a simple API in .NET Core that allows asynchronous processing of requests.

1. Request sent to Controller
2. Work scheduled on background service (IHostedService)
3. Controller returns 202
4. Background service performs long running operation

As the application will be running on IIS, a pool recycle can happen after the controller has returned a response. I would like to implement a way to allow the application to do a graceful shutdown - finishing it's currently executing tasks, but not accepting any new.

I am having difficulties having the graceful shutdown run to end. For testing purposes I've simplified the StopAsync() method, so now it only contains an async 20 second wait:

public async Task StopAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Hosted service is stopping...");

        try
        {
            // Signal cancellation to the executing method
            _stoppingTokenSource.Cancel();

            _logger.LogInformation("Delay by 20s");
            await Task.Delay(20000, cancellationToken);
            _logger.LogInformation("Delay over");
        }
        finally
        {
            // Await all pending download requests
            await Task.WhenAny(Task.WhenAll(_pendingTasks), Task.Delay(Timeout.Infinite, cancellationToken));

            _logger.LogInformation("Hosted service has been stopped successfully.");
        }
    }

The logs only show show:

• Hosted service is stopping...
• Delay by 20s

Looking at EventViewer I can see 2 events raised for the shutdown with exactly 10 seconds in between them:

• Sent shutdown HTTP message to process '11104' and received http status '202'.
• Failed to gracefully shutdown process '11104'.

I have already tried:

Settings up shutdown timeout on Program.cs

public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        WebHost.CreateDefaultBuilder(args)
            .UseStartup<Startup>()
            .UseShutdownTimeout(TimeSpan.FromMinutes(1));

Check Advanced Settings of application pool:

If you have done something similar, could you please let me know if I'm doing something wrong / point me in the right direction?

Thank you!

EDIT

While debugging the application, shutting down with CTRL + C produces the expected behaviour.

Answer 1

I've found a solution after having another look at the issue again.

It seems like ASP.NET Core Module in IIS have a separate shutdown time limit, which can be configured in the web.config file as:

    <configuration>
      <system.webServer>
        <handlers>
          <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
        </handlers>
        <aspNetCore
            shutdownTimeLimit="30"    - set timeout here in seconds
            processPath="dotnet"
            arguments=".\GracefulShutdown.dll"
            stdoutLogEnabled="false"
            stdoutLogFile=".\logs\stdout" />
      </system.webServer>
    </configuration>

Reference to settings at: https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/aspnet-core-module?view=aspnetcore-2.1

Ideally this should be available through code, which is still something I'm trying to resolve. If you find a way to do it, please share in comment.

Answer 2

In Configure(IApplicationBuilder app, IHostEnvironment env) function Register to ApplicationStopping in IHostApplicationLifetime ( Triggered when the application host is performing a graceful shutdown. Shutdown will block until this event completes)

  var applicationLifetime = app.ApplicationServices.GetRequiredService<IHostApplicationLifetime>();
  applicationLifetime.ApplicationStopping.Register(OnShutdown);

private void OnShutdown() { Write your code here that you want before shutdown, you can delay shutdown by Thread.Sleep(); }