Unity TlsException: Handshake failed UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED

Question

I'm trying to update my application's TcpClient to use TLS with SslStream instead of the normal Stream, the code i'm using for this seems to work outside of Unity, but fails when integrated in my Unity 2019.1.8 (tested on 2018 and 2017 as well) project.

To establish a connection and open a new SslStream I use the following code:

public static void InitClient(string hostName, int port, string certificateName)
{
    client = new TcpClient(hostName, port);

    if (client.Client.Connected)
    {
        Debug.LogFormat("Client connected succesfully");
    }
    else
    {
        Debug.LogErrorFormat("Client couldn't connect");
        return;
    }

    stream = new SslStream(client.GetStream(), false, new RemoteCertificateValidationCallback(ValidateServerCertificate), null);

    try
    {
        stream.AuthenticateAsClient(certificateName);
    }
    catch (AuthenticationException e)
    {
        Debug.LogErrorFormat("Error authenticating: {0}", e);
        if (e.InnerException != null)
        {
            Debug.LogErrorFormat("Inner exception: {0}", e);
        }
        Debug.LogErrorFormat("Authentication failed - closing connection");
        stream.Close();
        client.Close();
    }
}

And for validating the certificate

public static bool ValidateServerCertificate(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors)
{
    if (sslPolicyErrors == SslPolicyErrors.None)
        return true;

    Debug.LogErrorFormat("Certificate error: {0}", sslPolicyErrors);
    return false;
}

In Unity 2019.1.8 the client connects and will attempt to validate the remote certificate, which fails with the error TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR, verify result: UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED.

Making ValidateServerCertificate always return true lets my client connect without issue.

I tried replicating the issue in a standalone C# Console Application targeting .net framework 4.7.1 using the exact same code. Launching the client in this application will return true from ValidateServerCertificate from the sslPolicyErrors == SslPolicyErrors.None check.

I know that the certificate is a valid cert, issued by a trusted CA (as verified by the fact that the cert is accepted from a console app, and it having a padlock in browsers).

Why does the validation fail in Unity, but nowhere else?

Answer 1

Why Unity can't validate option 1:
Although the certificate is valid and correct (e.g. it works when using it in a web browser), it did not include the intermediate certificates chain to the root CA. This has as a result that no chain of trust can be formed (Unity doesn't cache/retrieve intermediates), resulting in the UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED flag being set.

To fix this I needed to append the certificate chain to my leaf certificate so that Unity can verify the entire chain up to the root CA. To find the certificate chain for your certificate you can use a "TLS certificate chain composer".

Depending on what software you use you may either need to include the chain in your certificate, or keep it in a seperate file. From whatsmychaincert (i'm in no way affiliated to this site, merely used it):

Note: some software requires you to put your site's certificate (e.g. example.com.crt) and your chain certificates (e.g. example.com.chain.crt) in separate files, while other software requires you to put your chain certificates after your site's certificate in the same file.

Why Unity can't validate option 2:
When your server's SSL certificate has expired Unity will throw the exact same UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED error without additional information that the certificate has been expired, so make sure the "Valid to" date is in the future (this would also cause the certificate to be denied when used in a web browser).

---

Why a browser/Console app can validate:
Software can have different implementation on how it deals with incomplete chains. It can either throw an error, stating that the chain is broken and can thus not be trusted (as is the case with Unity), or cache and save the intermediates for later use/retrieve it from previous sessions (as browsers and Microsoft's .net(core) do).
As explained in this answer (emphasis mine)

In general, SSL/TLS clients will try to validate the server certificate chain as received from the server. If that chain does not please the client, then the client's behaviour depends on the implementation: some clients simply give up; others (especially Windows/Internet Explorer) will try to build another chain using locally known intermediate CA and also downloading certificates from URL found in other certificates (the "authority information access" extension).

Not trusting the certificate when no chain is provided is a deliberate decision made by Unity due to cross-platform compatability, as stated in an [answer to issue 1115214][3]:

We may be able to solve this issue by doing a verification via the system specific TLS api instead of using OpenSSL/MbedTLS to validate against root certificates as we do today, however this solution would then not work cross-platform. So we don't want to implement it today, as it would hide the misconfigured server from the user on some but not all platforms.

---

I figured out this was a solution to my particular situation while asking this question, so decided to self-answer it for future references. However UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED can have all kind of causes, this just being one of them.

Answer 2

For anyone stumbling across this who's using UnityWebRequest and Let's Encrypt:

There is a known issue on Unity's Issue Tracker where you will find it's been fixed in only certain versions:

• Fixed in 2020.1.0a11
• Fixed in 2019.3.3f1
• Fixed in 2018.4.18f1

Changing Unity versions for an existing project is not a trivial task so we opted to buy a legit SSL cert which, although not cheap, was the simplest solution.