How can I use SslStream/AuthenticateAsServer with a self signed cert/key?

Question

I'd like to write (with C# and dotnet 4.8) a TCP service that uses TLS for security. To that end I'd like to use SslStream together with TcpListener, but I keep getting strange errors.

My experimental program is below. In brief:

• The program creates a self-signed PFX, based on code at https://stackoverflow.com/a/52535184/3568
• It launches a thread that will run the service while the main thread runs the client.
• The service thread listens for a connection.
• The client connects.
• The service sends a line to confirm the underlying TCP connection works.
• The client receives the line.
• The server and client both hand over control to SslStream, with the server using the PFX created earlier.
  • This is where the exception is thrown.
• Once TLS has been negotiated, the server sends another line, this time inside the TLS channel.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Security;
using System.Net.Sockets;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using System.Threading;

namespace TestTlsService
{
    class Program
    {
        static string tempCert;

        static void Main(string[] args)
        {
            /* Create a cert. */
            var cr = new CertificateRequest("cn=this.is.invalid", ECDsa.Create(), HashAlgorithmName.SHA256);
            using (var cert = cr.CreateSelfSigned(DateTime.UtcNow.AddDays(-1), DateTime.UtcNow.AddYears(+1)))
            {
                var exp = cert.Export(X509ContentType.Pfx);
                tempCert = Path.Combine(Path.GetTempPath(), "MySelfSignedCert.pfx");
                File.WriteAllBytes(tempCert, exp);
            }

            /* Launch a service thread. */
            Thread svc = new Thread(ServiceMain);
            svc.Start();
            Thread.Sleep(100);

            /* Connect as a client. */
            using (var tcp = new TcpClient())
            {
                tcp.Connect("localhost", 1984);
                var stream = tcp.GetStream();

                /* Read the line "GoTLS" from the server. */
                var insecureLine = ReadLine(stream);

                /* Hand over control to TLS. */
                using (var tls = new SslStream(stream, false, CheckCert))
                {
                    tls.AuthenticateAsClient("this.is.invalid");

                    /* Read a different line, this time securely. */
                    string line = ReadLine(tls);
                }
            }
        }

        static void ServiceMain()
        {
            /* Open a listener and start listening. */
            var listen = new TcpListener(IPAddress.Loopback, 1984);
            listen.Start();
            using (var tcp = listen.AcceptTcpClient())
            {
                /* Send "GoTLS" to the client insecurely. */
                var stream = tcp.GetStream();
                stream.Write(Encoding.ASCII.GetBytes("GoTLS\r\n"), 0, 7);

                /* Hand over control to TLS, using the self-signed cert from earlier. */
                using (var tls = new SslStream(stream))
                {
                    var cert = new X509Certificate2(tempCert);
                    tls.AuthenticateAsServer(cert);

                    /* Send a new message inside the secure channel. */
                    tls.Write(Encoding.ASCII.GetBytes("Hello\r\n"), 0, 7);
                }
            }
        }

        /* Simple function that just reads a line. */
        static string ReadLine(Stream stream)
        {
            byte[] line = new byte[100];
            int bytesIn = stream.Read(line, 0, 100);
            string lineAscii = Encoding.ASCII.GetString(line, 0, bytesIn);
            return lineAscii;
        }

        /* Accept all certificates. DO NOT COPY THIS INTO YOUR OWN CODE! */
        private static bool CheckCert(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
        {
            /* The full version will check the cert, but for now we'll just... */
            return true;
        }
    }
}

This program throws AuthenticationException in the AuthenticateAsServer line with the message "A call to SSPI failed, see inner exception." while the inner exception has "The client and server cannot communicate, because they do not possess a common algorithm". I find it a little unlikely that the SslStream library can't find a common protocol with itself.

My suspicion is that the root of my problem is that I'm using a self-signed certificate, as all of the examples of AuthenticateAsServer I can find use a pre-generated certificate file, which alas, I don't have.

How do I use SslStream on the server side to negotiate TLS with a self-signed cert?

---

To pre-empt anticipated questions...

I know the client side works because I've replaced "localhost" with a mail server (and skipped over the read-a-line) and the client thread works fine to read the secured welcome line that the remote mail server sends after TLS has been negotiated.

Setting ServicePointManager.SecurityProtocol to Tls12 doesn't help.

I get the same error if I remove the pre-SslStream "GoTLS" exchange.

I've searched for the error messages. All the answers I can find suggest setting the SecurityProtocol to Tls12, which I've already tried.

I've tried other formats of certificate instead of PFX, but I get different errors complaining that I'm missing the private key.

Answer 1

I don't know exactly why, but it doesn't like the ECDSA certificate you issue. If you instead use standard RSA:

var cr = new CertificateRequest(new X500DistinguishedName("cn=this.is.invalid"), RSA.Create(), HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);

and leaving everything else as is - then it will work as expected. In theory, TLS supports ECC (elliptic curve) keys, but I'm not expert in this and not sure why SslStream.AuthenticateAsServer doesn't like that certificate. Hopefully you don't need specifically ECC, then workaround above with RSA is perfectly fine.