Client certificate not getting added to the request (Certificate Verify)

Question

I'm trying to do a simple GET request to an external production server with a client certificate. They have added our certificate to their server, and I have successfully made requests through Postman (both the Chrome app and the Windows native app) and through standard browsers:

The Chrome app version of Postman uses the built-in certificate finder from Chrome. The native Postman app needs a .crt and a .key file, which I've extracted from my .p12 file.

In other words, the certificate is successfully found in the store, and also works when used from files (in a Windows native app, suggesting it should be possible in .NET).

---

Getting the certificate in C#

In my simple C# (.NET Framework 4.5.1) console application I am able to get the certificate from the store (or from files), and successfully use it to encrypt and decrypt a file (which I take it means I have full access to it from my application):

private static X509Certificate2 GetCertificate(string thumbprint)
{
    X509Store store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
    store.Open(OpenFlags.ReadOnly | OpenFlags.OpenExistingOnly);
    X509Certificate2Collection coll =
        store.Certificates.Find(X509FindType.FindByThumbprint, thumbprint,
            validOnly: true);
    X509Certificate2 certificate = coll.Count == 0 ? null : coll[0];
    return certificate;
}

---

Application code

I make the request to the server using either HttpClient or HttpWebRequest:

//A global setting to enable TLS1.2 which is disabled in .NET 4.5.1 and 4.5.2 by default,
//and disable SSL3 which has been deprecated for a while.
//The server I'm connecting to uses TLS1.2
ServicePointManager.SecurityProtocol &= ~SecurityProtocolType.Ssl3;
ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls11
    | SecurityProtocolType.Tls12;

X509Certificate cert = GetCertificate(thumbprint);
string url = "https://sapxi.example.com/XISOAPAdapter/MessageServlet";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.ClientCertificates.Add(cert);
request.Method = WebRequestMethods.Http.Get;

WebResponse basicResponse = request.GetResponse(); //This is where the exception is thrown
string responseString = new StreamReader(basicResponse.GetResponseStream()).ReadToEnd();

---

Exceptions

Both HttpClient or HttpWebRequest throws the same exceptions:

(WebException) The underlying connection was closed: An unexpected error occurred on a send.

(IOException) Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.

(SocketException) An existing connection was forcibly closed by the remote host

---

Tracing the request in Visual Studio

Enabling tracing, I get an output where both the certificate and private key is found (I've filtered out the verbose messages):

System.Net Error: 0 : [29136] Can't retrieve proxy settings for Uri 'https://sapxi.example.com/XISOAPAdapter/MessageServlet'. Error code: 12180.
System.Net Information: 0 : [29136] Associating HttpWebRequest#21454193 with ServicePoint#60068066
System.Net Information: 0 : [29136] Associating Connection#3741682 with HttpWebRequest#21454193
System.Net.Sockets Information: 0 : [29136] Socket#33675143 - Created connection from 192.168.168.177:56114 to 131.165.*.*:443.
System.Net Information: 0 : [29136] Connection#3741682 - Created connection from 192.168.168.177:56114 to 131.165.*.*:443.
System.Net Information: 0 : [29136] TlsStream#43332040::.ctor(host=sapxi.example.com, #certs=1)
System.Net Information: 0 : [29136] Associating HttpWebRequest#21454193 with ConnectStream#54444047
System.Net Information: 0 : [29136] HttpWebRequest#21454193 - Request: GET /XISOAPAdapter/MessageServlet HTTP/1.1

System.Net Information: 0 : [29136] ConnectStream#54444047 - Sending headers
{
Host: sapxi.example.com
Connection: Keep-Alive
}.
System.Net Information: 0 : [29136] SecureChannel#20234383::.ctor(hostname=sapxi.example.com, #clientCertificates=1, encryptionPolicy=RequireEncryption)
System.Net Information: 0 : [29136] Enumerating security packages:
System.Net Information: 0 : [29136]     Negotiate
System.Net Information: 0 : [29136]     NegoExtender
System.Net Information: 0 : [29136]     Kerberos
System.Net Information: 0 : [29136]     NTLM
System.Net Information: 0 : [29136]     TSSSP
System.Net Information: 0 : [29136]     pku2u
System.Net Information: 0 : [29136]     WDigest
System.Net Information: 0 : [29136]     Schannel
System.Net Information: 0 : [29136]     Microsoft Unified Security Protocol Provider
System.Net Information: 0 : [29136]     Default TLS SSP
System.Net Information: 0 : [29136]     CREDSSP
System.Net Information: 0 : [29136] SecureChannel#20234383 - Attempting to restart the session using the user-provided certificate:
*my certificate is here* (Issuer = CN=TRUST2408 OCES CA II, O=TRUST2408, C=DK)
System.Net Information: 0 : [29136] SecureChannel#20234383 - Left with 1 client certificates to choose from.
System.Net Information: 0 : [29136] SecureChannel#20234383 - Trying to find a matching certificate in the certificate store.
System.Net Information: 0 : [29136] SecureChannel#20234383 - Locating the private key for the certificate:
*my certificate is here*
System.Net Information: 0 : [29136] SecureChannel#20234383 - Certificate is of type X509Certificate2 and contains the private key.
System.Net Information: 0 : [29136] AcquireCredentialsHandle(package = Microsoft Unified Security Protocol Provider, intent  = Outbound, scc     = System.Net.SecureCredential)
System.Net Information: 0 : [29136] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = (null), targetName = sapxi.example.com, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Information: 0 : [29136] InitializeSecurityContext(In-Buffer length=0, Out-Buffer length=171, returned code=ContinueNeeded).
System.Net Information: 0 : [29136] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 278cca8:6d23888, targetName = sapxi.example.com, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Information: 0 : [29136] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=ContinueNeeded).
System.Net Information: 0 : [29136] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 278cca8:6d23888, targetName = sapxi.example.com, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Information: 0 : [29136] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=ContinueNeeded).
System.Net Information: 0 : [29136] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 278cca8:6d23888, targetName = sapxi.example.com, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Information: 0 : [29136] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=ContinueNeeded).
System.Net Information: 0 : [29136] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 278cca8:6d23888, targetName = sapxi.example.com, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Information: 0 : [29136] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=CredentialsNeeded).
System.Net Information: 0 : [29136] SecureChannel#20234383 - We have user-provided certificates. The server has specified 8 issuer(s). Looking for certificates that match any of the issuers.
System.Net Information: 0 : [29136] SecureChannel#20234383 - Selected certificate:
*my certificate is here*
System.Net Information: 0 : [29136] SecureChannel#20234383 - Left with 1 client certificates to choose from.
System.Net Information: 0 : [29136] SecureChannel#20234383 - Trying to find a matching certificate in the certificate store.
System.Net Information: 0 : [29136] SecureChannel#20234383 - Locating the private key for the certificate:
*my certificate is here*
System.Net Information: 0 : [29136] SecureChannel#20234383 - Certificate is of type X509Certificate2 and contains the private key.
System.Net Information: 0 : [29136] AcquireCredentialsHandle(package = Microsoft Unified Security Protocol Provider, intent  = Outbound, scc     = System.Net.SecureCredential)
System.Net Information: 0 : [29136] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 278cca8:6d23888, targetName = sapxi.example.com, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Information: 0 : [29136] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=349, returned code=ContinueNeeded).
System.Net.Sockets Error: 0 : [29136] Socket#33675143::UpdateStatusAfterSocketError() - ConnectionReset
System.Net.Sockets Error: 0 : [29136] Exception in Socket#33675143::Receive - An existing connection was forcibly closed by the remote host.
System.Net Error: 0 : [29136] Exception in HttpWebRequest#21454193:: - The underlying connection was closed: An unexpected error occurred on a send..

The above section is repeated once more and then it finally throws the exception chain. I've replaced the real URL and IP of the server with an example one.

The lines from

We have user-provided certificates. The server has specified 8 issuer(s). Looking for certificates that match any of the issuers.

to

Certificate is of type X509Certificate2 and contains the private key.

makes me think that the certificate is found correctly in HttpWebRequests's inner workings.

I can't tell what goes wrong from this output.

---

Debugging with Wireshark

In Wireshark I've compared Postman requests and my C# code and the only difference I see is that the Client Verify part (which includes the entire certificate) is not sent from C#, but it is sent via Postman (and browsers).

Via Postman and browsers, this is what it looks like:

And via C# this is what it looks like:

To me it looks like my application is ignoring the client certificate completely.

And when I don't provide the client certificate (//request.ClientCertificates.Add(cert)) I get exactly the same output in Wireshark, which seems to confirm this suspicion. In the tracing output in Visual Studio I just get Left with 0 client certificates to choose from. and no search for the certificate in the store or anything like that.

It's also worth noting that Wireshark makes it evident that Postman uses TLS1.2 successfully - and that my application code is also using TLS1.2.

---

Creating a local web page that requires a certificate

I got this to work, setting up the IIS Express to require certificates and then calling it. On the page I can see the certificate in the Request.ClientCertificates property. In wireshark, it doesn't send the Certificate Verify so something is still different.

But this page runs on my local machine, using the self-signed certificate that IIS Express prompted me to get installed. I have yet to set the project up on a production server with a valid certificate, and see if it behaves the same.

I'm not sure what this means exactly, but I think I can confirm that I'm not forgetting something basic, and that this is either an edge-case, or some protocol that the HttpWebRequest libraries in C# doesn't handle properly.

---

What else I've tried

• Adding the entire certificate chain/collection to the request (to my understanding, this is unnecessary, since the server has the issuer/CA to verify my certificate)
• Getting the certificate from a .key and .crt file, combining it in the code
• Using HttpClient's SendAsync() with a WebRequestHandler
• Targeting .NET Framework 4.6.1
• Targeting .NET Framework 4.7.1
• Enabling only TLS 1.2 (ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12)
• Adding the server's certificate to the request.ClientCertificates.Add(serverCert)
• Using WinHttpCertCfg.exe to grant access to the certificate, to the current user
• Running Visual Studio as admin
• Setting various settings on and off:

.

//Automatically verifying all server certificates (don't use this in production)
ServicePointManager.ServerCertificateValidationCallback =
    (sender, certificate, chain, sslPolicyErrors) =>
{
    return true; //This is not reached
};
ServicePointManager.Expect100Continue = true;
request.AllowAutoRedirect = true;
request.PreAuthenticate = true;
request.KeepAlive = false;
request.UserAgent = null;
request.CachePolicy = new HttpRequestCachePolicy(
    HttpCacheAgeControl.MaxAge, TimeSpan.FromSeconds(0));
//And several more that I didn't expect any effect from

If it helps, their server is running SAP XI, which is the application that denies me access. I don't know if that setup is very different to others, but since Postman is able to do the requests successfully, I don't suspect it to be very different. At worst it's just an above-average security protocol that still follows a standard.

The main idea I have is to setup the simple ASP page/API (that requires a client certificate) and put it on our production server. Another idea was to find an alternative to HttpClient. Or even worse, create my own, and just try copy the transaction flow that I see Postman do. But basically I'm running out of ideas. Any help is appreciated.

---

Question

If I must formulate a specific question, I think it'd be:

How can I make a GET request to a SAP XI server with my client certificate, using TLS 1.2 in C#?

---

Also, I'm not sure if I can reveal the URL or IP of the production server. Certainly none of you will be able to connect to it yourself either way, since they will not allow you to add your certificate to their server. So this won't be entirely reproducible I'm afraid. I guess there's no harm in revealing that the server belongs to KMD.

But if I can connect successfully to my own page/service and see the client-certificate there, then I think I will be past the goal post either way, so I think that's the way to go.

Sorry for the length of the question, but this way I've provided a lot of background research and details which should help answer'ers and future people diagnosing a very similar problem. I've tried to include some of the common issues in my question as well.

I'll of course answer this question myself when I figure it out, if this doesn't get any answers.

Thanks in advance.

Answer 1

While researching how to capture socket data to Wireshark, from my locally hosted page, I accidentally stumbled upon an article saying that "Certificate Verify" isn't sent over TLS 1.2 in "newer versions of Windows" (like Windows 10).

So I changed the protocol to TLS 1.0 and the request went through:

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;

With TLS 1.1 I get an exception, unlike what the guy in that article said:

(WebException) The request was aborted: Could not create SSL/TLS secure channel.

Why this worked isn't something I have time to investigate currently, as I'm already way behind schedule debugging this issue, but it sounds to me like a bug, much like another user claimed in another question.

I found a Microsoft article along these lines saying:

This issue only occurs with servers that downgrade the TLS session in an ungraceful way (such as by sending a TCP reset when receiving a TLS protocol version that the server does not support)

But since I start in TLS 1.2, and the server clearly accepts TLS 1.2 (via Postman and Chrome), it must be a tiny part of the TLS 1.2 protocol that isn't implemented the same way or something. I still don't understand how the Postman native Windows app manages to use TLS 1.2 though.

It may be worth noting that Internet Explorer first attempts TLS 1.2, and then after 2 resets (like my client), it just downgrades to TLS 1.0 and gets through. To me this sounds very similar to the update to Internet Explorer talked about in the article:

---

I realize this is not a great answer (when it comes to details of "why"), but at least it gives a hint as to what one might try if coming across similar issues.

If anyone understands this issue, and perhaps even knows how I can support TLS 1.2, then I'd appreciate it very much.