Scaling an IdentityServer4 service

Question

I have followed the IdentityServer4 quickstarts and am able to authenticate my javascript web page (almost the same as that provided in the quickstart) with a localhosted instance of IdentityServer using the Implicit grant. Again, my IdentityServer is almost exactly the same as that provided in the quickstart mentioned above - it just has some custom user details.

I then moved my application (C# .NET Core) into a docker container and have hosted one instance of this within a Kubernetes cluster (single instance) and created a Kubernetes service (facade over one or more 'real' services) which lets me access the identity server from outside the cluster. I can modify my JavaScript web page and point it at my Kubernetes service and it will still quite happily show the login page and it seems to work as expected.

When I then scale the IdentityServer to three instances (all served behind a single Kubernetes service), I start running into problems. The Kubernetes service round-robins requests to each identity server, so the first will display the login page, but the second will try and handle the authentication after I press the login button. This results in the following error:

System.InvalidOperationException: The antiforgery token could not be decrypted. ---> System.Security.Cryptography.CryptographicException: The key {19742e88-9dc6-44a0-9e89-e7b09db83329} was not found in the key ring. at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.DangerousUnprotect(Byte[] protectedData, Boolean ignoreRevocationErrors, Boolean& requiresMigration, Boolean& wasRevoked) at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Unprotect(Byte[] protectedData) at Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgeryTokenSerializer.Deserialize(String serializedToken) --- End of inner exception stack trace --- ... And lots more......

So - I understand that I am getting this error because the expectation is that the same IdentityServer should service the request for a page that it has shown (otherwise how would the anti-forgery token work, right?), but what I am trying to understand is how I can make this work in a replicated environment.

I don't want to host multiple identity servers on different IP's/ports; I'm trying to build a HA configuration where if one IdentityServer dies, nothing calling the endpoint should care (because requests should be serviced by other working instances).

I said i was using the quickstart code - this means that in the startup of the IdentityServer, there is code that looks like this...

    public void ConfigureServices(IServiceCollection services)  
    {
        services.AddMvc();

        services.AddIdentityServer(options =>
            {
                options.Events.RaiseSuccessEvents = true;
                options.Events.RaiseFailureEvents = true;
                options.Events.RaiseErrorEvents = true;
            })
            .AddTemporarySigningCredential()
            .AddInMemoryIdentityResources(Config.GetIdentityResources())
            .AddInMemoryApiResources(Config.GetApiResources())
            .AddInMemoryClients(Config.GetClients())

I am assuming that I need to replace the .AddTemporarySigningCredential() logic with a certificate that can be used by all instances of the IdentityServer which are running in my Kubernetes cluster. Not knowing how MVC really works (MVC6 is used to generate the login pages in the IdentityServer service, which I got from the example code - link above) - I want to know if just changing the code to use a proper certificate which is shared between all services will be sufficient to get a prototype HA IdentityServer cluster working?

By working, I mean that my expectation is that I can have n number of IdentityServer instances running in a Kubernetes cluster, have a Kubernetes service to act as a facade over however many IdentityServer's I have running, and be able to authenticate using multiple IdentityServer instances which can share data to the extent that they all provide exactly the same authority to my calling web applications, and can handle each other's requests in the event that one or more instances die.

Any help or insight would be appreciated.

Answer 1

I think I have worked this out. To resolve my issue, I have done two things:

1. Create my own X509 certificate and shared this certificate between each of my IdentityServer's. There are lots of examples of how to create valid certificates on the net; I just used

   services.AddIdentityServer(...).AddSigningCredential(new X509Certificate2(bytes, "password")
   

   in my startup class.

2. Dug into the MVC framework code and worked out that I needed to implement a Key storage provider in order to share state between different instances of the MVC part of Identity Server which serves up the login page.

It turns out that there is a Redis backed KSP available from NuGet, which means that I just need to spin up a private redis instance in my Kube cluster (which isn't accessible outside of my cluster) to share decryption secrets.

/* Note: Use an IP, or resolve from DNS prior to adding redis based key store as direct DNS resolution doesn't work for this inside a K8s cluster, though it works quite happily in a Windows environment. */  
var redis = ConnectionMultiplexer.Connect("1.2.3.4:6379");
services.AddDataProtection()
        .PersistKeysToRedis(redis, "DataProtection-Keys");

I can now scale my identity service to 3 instances and have a Kube service acting as a facade over all the available instances. I can watch the logs as Kubernetes round-robin's requests between the identity service, and my authentication happens just as I expect.

Thanks to those who commented on the question prior to this post.

Answer 2

For those using Kubernetes, it's possible to use the File System Key Storage Provider

public void ConfigureServices(IServiceCollection services)
{
    services.AddDataProtection()
        .PersistKeysToFileSystem(new DirectoryInfo(@"/app/key-storage"));
}

where the directory '/app/key-storage' is mapped to an nfs backed persistent volume.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-key-storage
spec:
  selector:
    matchLabels:
      type: nfs-pv
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Mi

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv
  labels:
    type: nfs-pv
spec:
  storageClassName: manual
  capacity:
    storage: 10Mi
  accessModes:
    - ReadWriteMany
  nfs:
    server: <server>
    path: /<path>
  persistentVolumeReclaimPolicy: Delete

And in the IDP Deployment

template:
  spec:
    containers:
      - name: <name>
        volumeMounts:
          - name: key-storage
            mountPath: /app/key-storage
            readOnly: false
    volumes:
      - name: key-storage
        persistentVolumeClaim:
        claimName: pvc-key-storage

And you need the signing cert. This can added as a secret, and then the IDP Deployment can use another volume to mount the secret (not shown).

apiVersion: v1
kind: Secret
metadata:
  name: cert-secret
  labels:
    app: <app-label>
type: Opaque
data:
  signingcert.pfx: <base64 cert value>