Multi-endpoint Kestrel configuration with different TLS certificates serves the wrong certificate

Question

I am trying to set up the Kestrel server in an ASP.NET Core MVC-based service app in Linux environment through its appsettings.json config file so that it has exactly two endpoints, both using HTTPS, but each with a different certificate. One endpoint listens on 0.0.0.0, the other only on localhost. The service serves as the host for IdentityServer4. The intended runtime environment for the server will be a Docker container, which is a part of a multiservice-based system, currently run as a docker-compose cluster.

I have studied the official documentation at https://learn.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel?view=aspnetcore-2.2#endpoint-configuration and assumed it should be easily achievable. I have created a self-signed root CA certificate and generated two certificates signed by the root. One of them has the Common Name set to the name the container will have in the cluster and is intended for secure communication with the external environment. The other certificate has the CN set to localhost and is used because the server has multiple controllers that will be available from outside and they all will transform incoming requests and forward to a special internal controller that will handle them. (I realize that is a little paranoid of me, but that's beside the point.)

I have tried to configure Kestrel so that the main endpoint uses the default certificate and the localhost endpoint uses the ad-hoc certificate, i.e. like so:

  "Kestrel": {
    "Endpoints": {
      "Https": {
        "Url": "https://+:2051"
      },
      "HttpsLocalhost": {
        "Url": "https://localhost:2052",
        "Certificate:": {
          "Path": "path/to/localhost-dev.pfx",
          "Password": "password"
        }
      }
    },
    "Certificates": {
      "Default": {
        "Path": "path/to/identity-api-dev.pfx",
        "Password": "password"
      }
    }
  }

which appears to be the correct configuration as per the docs, which say that the server requires a default certificate to be defined if configuring HTTPS explicitly.

The outward facing controller uses a lazily initialized HttpClient, whose BaseAddress property is initialized on the first received request with the URL of the localhost endpoint. That URL is determined trivially with this method

private string GetIdentityServiceHostAndPort()
{
    IServerAddressesFeature addresses = Server.Features.Get<IServerAddressesFeature>();

    // Get the first that uses HTTPS and is on localhost
    foreach (string url in addresses.Addresses)
    {
        if (url.StartsWith("https://localhost"))
        {
            return url;
        }
    }

    throw new ArgumentException("Could not determine identity service's host/port, is it listening on localhost using HTTPS?");
}

The address returned by the above method is then passed to the following method that initializes the controller's HttpClient instance with it and also queries the IS4's OpenID configuration endpoint for the discovery document and uses that to further initialize the controller:

private static async Task InitializeTokenEndpointHttpClientAsync(string baseAddress)
{
    TokenEndpointClient = new HttpClient();
    TokenEndpointClient.BaseAddress = new Uri(baseAddress);

    HttpResponseMessage response = await TokenEndpointClient.GetAsync(".well-known/openid-configuration");
    string content = await response.Content.ReadAsStringAsync();
    if (response.StatusCode != HttpStatusCode.OK)
    {
        throw new InvalidOperationException("Error querying IdentityServer for discovery document: " + content);
    }

    DiscoveryDocument document = JsonConvert.DeserializeObject<DiscoveryDocument>(content);
    Uri fullTokenEndpointUri = new Uri(document.token_endpoint);
    TokenEndpointPath = fullTokenEndpointUri.LocalPath;
}

Since I have already spent a sizable amount of time trying to debug this issue, I have made sure that the server is configured to listen both on any address with one port and on localhost with another. The GetIdentityServiceHostAndPort() method does indeed return the correct URL.

Now, the problem I am facing is that the request for the discovery document in the method above dies with

System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.

Obviously, the certificate validation process failed, which seems suspicious to me, since the request was supposed to go to the endpoint that does have a certificate with CN=localhost set.

When I tried manually requesting the discovery document with

curl -g 'https://localhost:2052/.well-known/openid-configuration'

I got this message

curl: (60) SSL: certificate subject name 'identity.api' does not match target host name 'localhost'

I do not understand why the server would try to send this certificate. I know it is the configured default, but there is also the certificate expressly defined for the localhost endpoint and I would expect that one to be used in this case.

I am sure I am missing something but as I said, I have already spent several hours trying to find the culprit, so I will be very grateful for any input.

Answer 1

Just to make this clear and embarrass myself in front of the entire developer world at the same time, the issue was that I was constantly overlooking a typo in the Kestrel configuration section. There is an extraneous colon in the Certificate key string in the HttpsLocalhost subsection, which I could not spot if my life depended on it.

Thankfully, it did not, and of course the configuration worked after this fix.