Deploying .net Core 3 linux container on azure web app container with IdentityServer4 certification/http error

Question

I am trying to use the .Net Core Clean Architecture App Template and get it running in containers and deployed through an azure CI/CD pipeline

I have the containerized version of the template running locally in linux container with port 5001 and everything works perfectly.

I have the azure pipeline build process working properly and it creats image in my container registry.

The problem is once I deploy/release to a Web App for Containers, the app fails and throws the following error:

Application startup exception System.InvalidOperationException: Couldn't find a valid certificate with subject 'CN=localhost' on the 'CurrentUser\My' at Microsoft.AspNetCore.ApiAuthorization.IdentityServer.SigningKeysLoader.LoadFromStoreCert(String subject, String storeName, StoreLocation storeLocation, DateTimeOffset currentTime)

What I have done:

1. Following these docs from MS I have created a local dev cert:

   dotnet dev-certs https -ep %USERPROFILE%\.aspnet\https\aspnetapp.pfx -p { password here }

   dotnet dev-certs https --trust

2. I then imported this into the Web App as a private .pfx cert.

3. I added an application setting WEBSITE_LOAD_CERTIFICATES with the "thumb" value of the cert

4. I used the "hostname" of the imported cert in the Identity Server appSettings.json section (hostname=localhost in my case)

When the Web app loads, it shows :( Application error and the docker logs give me the error I quoted above.

I am pretty sure this is related to the Identity server set up and the appSettings.json values here:

  "IdentityServer": {
    "Key": {
      "Type": "Store",
      "StoreName": "My",
      "StoreLocation": "CurrentUser",
      "Name": "CN=localhost"
    }
  }

Can someone help me figure out how to resolve this error?

EDIT 1 - Manually specify file for IdentityServer Key

This is related to identity server for sure. I tried to manually set the Cert as a file in the appSettings.json like this:

  "IdentityServer": {
    "Key": {
      "Type": "File",
      "FilePath": "aspnetapp.pfx",
      "Password": "Your_password123"
    }
  }

Now I get this error:

Loading certificate file at '/app/aspnetapp.pfx' with storage flags ''. Application startup exception System.InvalidOperationException: There was an error loading the certificate. The file '/app/aspnetapp.pfx' was not found. Microsoft.AspNetCore.ApiAuthorization.IdentityServer.SigningKeysLoader.LoadFromFile

I added this to the dockerfile:

WORKDIR /app
COPY ["/aspnetapp.pfx", "/app"]
RUN find /app

And as you can see from the image below, the files are showing in the build directory for the app:

I also made sure that the aspnetapp.pfx is not getting ignored by the .gitignore or .dockerignore files.

I cannot figure out why it won't load this file. It appears like it exists right where it is supposed to.

EDIT 2 using cert thumb and updated path

So I used tnc1977 suggestion and had this as my setting for the identity key

  "IdentityServer": {
    "Key": {
      "Type": "File",
      "FilePath": "/var/ssl/private/<thumb_value>.p12",
      "Password": "Your_password123"
    }
  }

However, this gave another error:

There was an error loading the certificate. Either the password is incorrect or the process does not have permisions to store the key in the Keyset 'EphemeralKeySet' Interop+Crypto+OpenSslCryptographicException: error:23076071:PKCS12 routines:PKCS12_parse:mac verify failure

EDIT 3: Valid Azure App Certificate

I purchased an Azure App Certificate and added a custom domain with TSL set up and the same errors appear

EDIT 4: Load Cert in code startup.cs - new error:

I now know that I caanot use the cert store CurrentUser/My because that is for windows. Linux containers have to manually load the cert in code.

I am using the thumbprint of aa application certificate that has been added to the azure web app. It is a private azure app cert and it has been verified against a custom domain.

I added this code to my statup.cs configureservices (I know hardcoding these values is not best practice but I want to just see if it could load the cert, I will wsitch to env variables and key vault):

        // linux file path for private keys
        var cryptBytes = File.ReadAllBytes("/var/ssl/private/<thumbprint>.p12");
        var cert = new X509Certificate2(cryptBytes, "");

        services.AddIdentityServer().AddSigningCredential(cert);

I enter a blank password because I think that is what you are supposed to do. I am now getting the following error in my docker logs which leads me to believe the cert loaded and now the error is related to me using both services.AddIdentityServer().AddSigningCredential(cert); in startup.cs configureservices and app.UseIdentityServer() in startup.cs configure:

Unhandled exception. System.InvalidOperationException: Decorator already registered for type: IAuthenticationService.

I am not sure how to add the cert to the app.UseIdentityServer(); line.

EDIT 5

after a lot more digging, unfortunately @tnc1997 answer will not work. IN asp.net core 3 calls to app.UseIdentityServer in my satrtup.cs internally reverence a method that will look for the identity server Key,File,Pass etc in the appsetting(environment).json file.

As a result, even if I loaded the cert in code like tnc1997 shows, the application still looks in the settings file. So the settings file has to contain the corect details for the IS4 key.

Also, azure does not place the cert in the typical trusted location in the linux container. From what I have read, it appears that the only way to do this is to mount a volume (in this case an azure storage file share) and use the cert uploaded to that file share.

I can confirm that this works locally, but now I am still having issues running the container, the front end loads and it appears that the web api project does not start. I am going to post another question to address that issue.

Answer 1

Original Answer

I think the problem could be that you are attempting to load a certificate in a Linux container using the Windows certificate store.

The documentation here gives a good overview regarding how you can use an app service private certificate in a Linux hosted app:

1. In the Azure portal, from the left menu, select App Services > <app-name>.
2. From the left navigation of your app, select TLS/SSL settings, then select Private Key Certificates (.pfx) or Public Key Certificates (.cer).
3. Find the certificate you want to use and copy the thumbprint.
4. To access a certificate in your app code, add its thumbprint to the WEBSITE_LOAD_CERTIFICATES app setting.
5. The WEBSITE_LOAD_CERTIFICATES app setting makes the specified certificate accessible to your Linux hosted apps (including custom container apps) as files. The files are found under the following directories:
   • Private certificates - /var/ssl/private (.p12 files)
   • Public certificates - /var/ssl/certs (.der files)
6. Use the code sample below to load the specified certificate into your Linux hosted apps (including custom container apps):

   using System;
   using System.IO;
   using System.Security.Cryptography.X509Certificates;
   
   var bytes = File.ReadAllBytes($"/var/ssl/private/{Configuration["WEBSITE_LOAD_CERTIFICATES"]}.p12");
   var cert = new X509Certificate2(bytes);
   

Signing Credentials

Here are the steps that I used to generate signing credentials:

1. Install OpenSSL.
2. Generate private key and public certificate.
   1. Run openssl req -x509 -newkey rsa:4096 -sha256 -nodes -keyout example.com.key -out example.com.crt -subj "/CN=example.com" -days 365 replacing example.com with name of the site.
3. Combine the above into a single PFX file.
   1. Run openssl pkcs12 -export -out example.com.pfx -inkey example.com.key -in example.com.crt replacing example.com with the name of the site.
4. Upload the PFX file to Azure.
   1. In the Azure portal, from the left menu, select App Services > <app-name>.
   2. From the left navigation of your app, select TLS/SSL settings, then select Private Key Certificates (.pfx), then upload the above PFX file.
5. Configure app settings.
   1. Add the thumbprint of the PFX file above to the WEBSITE_LOAD_CERTIFICATES app setting in the App Service.

IdentityServer

The below code sample shows a complete Startup.cs configuration which could be used to get an IdentityServer application up and running:

namespace IdentityServer
{
    public class Startup
    {
        public Startup(IConfiguration configuration, IWebHostEnvironment environment)
        {
            Configuration = configuration;
            Environment = environment;
        }

        public IConfiguration Configuration { get; }

        public IWebHostEnvironment Environment { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        // For more information on how to configure your application, visit https://go.microsoft.com/fwlink/?LinkID=398940
        public void ConfigureServices(IServiceCollection services)
        {
            void ConfigureDbContext(DbContextOptionsBuilder builder)
            {
                builder.UseNpgsql(Configuration.GetConnectionString("DefaultConnection"));
            }

            var builder = services.AddIdentityServer()
                .AddConfigurationStore(options => { options.ConfigureDbContext = ConfigureDbContext; })
                .AddOperationalStore(options => { options.ConfigureDbContext = ConfigureDbContext; });

            if (Environment.IsDevelopment())
            {
                builder.AddDeveloperSigningCredential();
            }
            else
            {
                try
                {
                    var bytes = File.ReadAllBytes($"/var/ssl/private/{Configuration["WEBSITE_LOAD_CERTIFICATES"]}.p12");
                    var certificate = new X509Certificate2(bytes);
                    builder.AddSigningCredential(certificate);
                }
                catch (FileNotFoundException)
                {
                    throw new Exception($"The certificate with the thumbprint \"{Configuration["WEBSITE_LOAD_CERTIFICATES"].Substring(0, 8)}...\" could not be found.");
                }
            }
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment()) app.UseDeveloperExceptionPage();

            app.UseIdentityServer();
        }
    }
}

Clean Architecture

The below code sample shows a complete DependencyInjection.cs configuration which could be used to get a Clean Architecture application up and running:

namespace CleanArchitecture.Infrastructure
{
    public static class DependencyInjection
    {
        public static IServiceCollection AddInfrastructure(this IServiceCollection services, IConfiguration configuration)
        {
            void ConfigureDbContext(DbContextOptionsBuilder builder)
            {
                if (configuration.GetValue<bool>("UseInMemoryDatabase"))
                {
                    builder.UseInMemoryDatabase("CleanArchitectureDb");
                }
                else
                {
                    builder.UseSqlServer(configuration.GetConnectionString("DefaultConnection"), b => b.MigrationsAssembly(typeof(ApplicationDbContext).Assembly.FullName));
                }
            }

            services.AddDbContext<ApplicationDbContext>(ConfigureDbContext);

            services.AddScoped<IApplicationDbContext>(provider => provider.GetService<ApplicationDbContext>());

            services.AddScoped<IDomainEventService, DomainEventService>();

            services.AddDefaultIdentity<ApplicationUser>()
                .AddEntityFrameworkStores<ApplicationDbContext>();

            var builder = services.AddIdentityServer()
                .AddConfigurationStore(options => { options.ConfigureDbContext = ConfigureDbContext; })
                .AddOperationalStore(options => { options.ConfigureDbContext = ConfigureDbContext; })
                .AddAspNetIdentity<ApplicationUser>();

            var bytes = File.ReadAllBytes($"/var/ssl/private/{Configuration["WEBSITE_LOAD_CERTIFICATES"]}.p12");
            var certificate = new X509Certificate2(bytes);
            builder.AddSigningCredential(certificate);

            services.AddTransient<IDateTime, DateTimeService>();
            services.AddTransient<IIdentityService, IdentityService>();
            services.AddTransient<ICsvFileBuilder, CsvFileBuilder>();

            services.AddAuthentication()
                .AddIdentityServerJwt();

            return services;
        }
    }
}

Answer 2

I think the problem is that your application in the container does not trust the locally created developer certificate. Its something you can only use on your machine because a dev root certificate is installed on your computer.

The container will never trust the certificate created by dotnet dev-certs.

You need to get a properly trusted certificate, for example from LetsEncrypt.