SSL Error "The message received was unexpected or badly formatted" for a .NET application on one specific machine only

Question

I have a .NET Core 3.1 C# application which is calling an API via HTTPS (and presenting its public key as part of getting the token as that certificate is later used to decrypt information sent back separately). On just about all our machines, it is working, but on one Windows 8.1 machine, we get the following series of exceptions when we try to initially connect for an authentication token:

The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
---> System.ComponentModel.Win32Exception (0x80090326): The message received was unexpected or badly formatted.

The exception is thrown from System.Net.Http.HttpClient.FinishSendAsyncBuffered so I suspect it is happening at the HTTPS level and our certificate stuff is not really relevant here anyway.

Our code to get the token looks like this:

The constructor for the auth service:

  public XXXXAuthService(IXXDbService dbService, XXXXApiConfig config)
        {
            _dbService = dbService;
            _config = config;
            
            // try forcing TLS1.2 for SSL connection exceptions thrown in some operating environments
            ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
            
            _httpClient = new HttpClient {BaseAddress = new Uri(config.BaseUrl)};
            _httpClient.DefaultRequestHeaders.Accept.Clear();
            _httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
        }

Code to get the auth token:

private async Task<string> GetXXXXBearerToken(string userId, DateTime creationTime)
        {
            var token = await GenerateProviderJwtForXXXX(userId, creationTime);
            var kvp = new List<KeyValuePair<string, string>>
            {
                new KeyValuePair<string, string>("grant_type", "urn:ietf:params:oauth:grant-type:token-exchange"),
                new KeyValuePair<string, string>("subject_token", token),
                new KeyValuePair<string, string>("subject_token_type", "urn:ietf:params:oauth:token-type:jwt")
            };
            var data = new FormUrlEncodedContent(kvp);
            var publicKey = await GetXXXXPublicKey();

            _httpClient.DefaultRequestHeaders.Remove("X-XXXX-Public-Cert");
            _httpClient.DefaultRequestHeaders.Add("X-XXXX-Public-Cert", publicKey);

            var response = await _httpClient.PostAsync("Identity/token", data);
            if (!response.IsSuccessStatusCode)
                throw new Exception("XXXX Token Server Error: " + response.ReasonPhrase);
            var result = await response.Content.ReadAsStringAsync();

            var authResponse = JsonConvert.DeserializeObject<OAuthResponse>(result);

            if (!string.IsNullOrEmpty(authResponse.access_token))
                return authResponse.access_token;

            System.Diagnostics.Trace.WriteLine("Token Exchange Result: " + result);
            if (!string.IsNullOrEmpty(authResponse.error))
            {
                var outcome = new XXX.XXXX.Model.OperationOutcome();
                outcome.Issue.Add(new XXX.XXXX.Model.OperationOutcome.IssueComponent()
                {
                    //some code to throw an error is here
            }

            throw new XXX.XXXX.Rest.XXXXOperationException("Bearer Token Exchange failed", response.StatusCode);
        }

Unfortunately none of the existing questions/advice anywhere on Stack Overflow, or the rest of the web, for this particular error seems to have helped. They are primarily about version discrepancies between client and server which seems not to be the case here as I am forcing TLS 1.2 (which is active and enabled on the failing machine).

Interestingly, I can visit the server URL in a browser via HTTPS just fine, which suggests there is something about my code that is the problem rather than the machine, but it works everywhere else.

I have confirmed that:

• The certificate I am using to authenticate the connection on the machine is valid and has a chain of trust (though as above I don't think we are getting that far as the TLS connection itself is failing)
• The server we are calling supports TLS 1.2 (by forcing it)
• I can get to the website for the URL independently via the browser

Is there something I need to do either in the code or on the machine to get this call to work everywhere?

Things I have tried to resolve the issue

• Installing all Windows 8.1 updates to present day
• Forcing TLS 1.2 in the code (see above code sample)
• Limiting VM to TLS 1.2 only

Answer 1

I might be able to at least point you in the right direction…

Same Symptoms
I had a .NET Core 3.1 web app running on IIS (Windows Server 2012 R2) that got the exact same error and stacktrace when it tried to connect to another server using TLS 1.2. I also had the symptom where I could connect with the browser (Chrome), but not with the app. (Would have been interesting to see if Internet Explorer browser worked though.)

Root Cause
The TLS handshake was failing because the two servers were unable to agree on a common cipher suite. (Using Wireshark, I discovered that when my app tried to connect it provided a more limited set of cipher suites than when the Chrome browser made the call.)

Solution
In my case, I used IIS Crypto (a small free tool: https://www.nartac.com/Products/IISCrypto/) to enable additional cipher suites on my web app's server. I downloaded and ran IIS Crypto, checkmarked additional cipher suites on its Cipher Suites tab, and then restarted the machine.

One of the new cipher suites worked with my app and the destination server, so the TLS handshake was successful and the error was resolved.

One quick caveat: Some cipher suites are more secure than others, so you'll want to read up on best practices.

Addendum
If you want to further diagnose the failure, I'd recommend installing Wireshark (another free tool: https://www.wireshark.org/#download) on the machine with your .NET Core app. If a TLS Handshake Failure is the issue, you will see a message like: Alert (Level: Fatal, Description: Handshake Failure)

This primer on wireshark output helped me: https://blog.catchpoint.com/2017/05/12/dissecting-tls-using-wireshark/

Answer 2

I faced a simular issue, and in order to help others here's what I concluded:

Sucessfully executing this code doesn't mean that your application supports the specified protocol version, and the "SSL Error" can still occur later on when trying to establish a connection:

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

In my case I was trying to force Tls13 and found out that my app configuration didn't actually support it:

• net core 3.0 running on a Windows Server Datacenter 2019, version 1809

So I had to change my configuration to the following which provides support for the protocol version I needed:

• Net framework 5.0 on a Windows Server Datacenter 2022, OS build 20348.288

I was trying to connect to an endpoint that suddenly dropped Tls 1.2 support (not sure why) and from then on only accepted Tls 1.3.