Running multiple instances of Identity server in docker cluster

Question

I have a .net core 2.0 application basically Identity server 4. This runs just fine when I have only one instance. However if I try to run more then one instance of the identity server I start having issues.

First issue

An unhandled exception occurred while processing the request. CryptographicException: The key {ec55dd66-7caf-4423-9dd6-74768e80675d} was not found in the key ring. Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, bool allowOperationsOnRevokedKeys, out UnprotectStatus status)

InvalidOperationException: The antiforgery token could not be decrypted. Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgeryTokenSerializer.Deserialize(string serializedToken)

I was able to work out that this is because the key was being generated on all the instances of the identity server rather than just one key generated and they all used that.

I added the following code.

services.AddDataProtection()
            .PersistKeysToFileSystem(new DirectoryInfo(settingsSetup.Settings.PersistKeysDirectory))
            .SetDefaultKeyLifetime(TimeSpan.FromDays(90))
            .SetApplicationName($"Awesome-IdentityServer-{_env.EnvironmentName}"); 

Which basically tell the identity servers where to store the key. I followed the instructions found here Persisting keys when hosting in a Docker container so I have A folder that's a Docker volume that persists beyond the container's lifetime, such as a shared volume or a host-mounted volume.

Unforutnatlly this hasnt worked either i am now getting the following error

"idsrv was not authenticated. Failure message: Unprotect ticket failed"

Which i believe means that the keys need to be encrypted somehow.

What is Unprotect ticket failed and how do I solve it? Can i run multiple instances of Identity server in docker nodes?

Now with encryption.

services.AddDataProtection()
            .PersistKeysToFileSystem(new DirectoryInfo(settingsSetup.Settings.PersistKeysDirectory))
            .SetDefaultKeyLifetime(TimeSpan.FromDays(90))
            .ProtectKeysWithCertificate(LoadCertificate())
            .SetApplicationName($"Awesome-IdentityServer-{_env.EnvironmentName}");

Identity server responds with the following error in the logs.

No XML encryptor configured. Key {2e0f629c-9dca-44fa-922e-5c5613bd27c8} may be persisted to storage in unencrypted form.

The user is shown this error

CryptographicException: Unable to retrieve the decryption key. System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData encryptedData, string symmetricAlgorithmUri)

Authentication with Docker in ASP.NET Core also mentions that this should be working.

I originally thought this was an issue with Identity Server 4 since their documentation stated that it is stateless. I posted an issue over on GitHub Stateless or not Stateless after going back and forth with them I am inclined to think this is more of a docker issue then an Identity server issue.

Answer 1

The usual issue I've seen with clients is that they've put their AddDataProtection registration before AddIdentityServer.

AddIdentityServer also calls AddDataProtection with the default settings, which unfortunately will override whatever registrations you previously made.

So short answer: you need to put AddDataProtection after AddIdentityServer in your ConfigureServices method.