Kestrel throwing "Invalid Host header" error while serving traffic

Question

We have recently migrated an ASP.NET MVC 5 application to ASP.NET Core 2.2.

Everything seems to be working fine, however we are getting the following exception on a fairly regular basis (about three times a second, more on that in a moment):

BadHttpRequestException: Invalid Host header: '~^appname.*$'
  Module "Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException", line 0, col 0, in Throw
    Void Throw(Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.RequestRejectionReason, System.String)
  Module "Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection", line 95, col 0, in EnsureHostHeaderExists
    Void EnsureHostHeaderExists()
  Module "Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection", line 196, col 0, in TryParseRequest
    Boolean TryParseRequest(System.IO.Pipelines.ReadResult, Boolean ByRef)
  Module "Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__185`1", line 170, col 0, in MoveNext
    Void MoveNext()
  Module "System.Runtime.ExceptionServices.ExceptionDispatchInfo", line 12, col 0, in Throw
    Void Throw()
  Module "System.Runtime.CompilerServices.TaskAwaiter", line 46, col 0, in HandleNonSuccessAndDebuggerNotification
    Void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
  Module "Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequestsAsync>d__184`1", line 135, col 0, in MoveNext
    Void MoveNext()

In consulting with a member of our ops team it's clear that we have three instances of HAProxy all checking each node of our application about once a second.

Every request to this application would flow as follows:

HAProxy -> Nginx -> Kestrel/ASP.NET Core application

My question is, how can I determine what is going on here?

Answer 1

The current version of Kestrel (we're using ASP.NET Core 2.2) does not support HTTP access logging.

See this issue: Support access logging with Common Log Format/Extended Log Format

In order to track down the issue I had to have our ops team review our nginx logs (nginx is the reverse proxy we're using) to determine where requests with the invalid host header were coming from.

The issue turned out to be a mal-configured status checking application.