Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

Java 11 and 12 SSL sockets fail on a handshake_failure error with TLSv1.3 enabled

Tags:

java

ssl

sslhandshakeexception

tls1.3

While developing a SSL-based (non HTTP) server in Java 12, I met unexpected difficulties for having the server and the client talk together. The connection was always interrupted with a handshake_failure error. After lots of efforts (initially I was incriminating my certificates), I was able to isolate the issue thanks to a HelloWorld SSL client/server pair at the socket level: the handshake failure was caused by the presence of TLSv1.3 among the protocols enabled by default in the JDK.

So far, my solution to avoid the handshake error is to disable TLSv1.3 completely (server-side), but I am not satisfied by this solution. I would like to understand what's going on (assuming it's not a bug in the JDK or in the system but in my application or my environment).

I provide some code below. Can anyone check and tell me what I am doing wrong? Thanks and best regards.

Here is my configuration: OSX 10.14 (Mojave) I have tested with the following versions of Java:

EDIT: I added Java 11 to my tests and the problem occurs also for this version

  • Java 8 (oracle64-1.8.0.181): no handshake failure
  • Java 10 (oracle64-10.0.2 2018-07-17): no handshake failure
  • Java 11 (openjdk version "11.0.4" 2019-07-16): handshake failure
  • Java 12 (oracle64-12.0.1 2019-04-16): handshake failure
  • Java 12 (openjdk 12.0.2 2019-07-16): handshake failure
// HelloSSLServer.java

import java.net.*;
import javax.net.*;
import javax.net.ssl.*;
import java.io.*;

public class HelloSSLServer {

    public static void main(String args[]) throws Exception {
        int port = 1234;
        boolean needClientAuth = false;
        ServerSocketFactory factory = SSLServerSocketFactory.getDefault();
        try (ServerSocket ss = factory.createServerSocket(port)) {
            SSLServerSocket ssl = (SSLServerSocket) ss;
            ssl.setNeedClientAuth(needClientAuth);
            ssl.setEnabledProtocols(new String[] { "TLSv1.2" }); // fails with TLSv1.3 (or with the default, i.e. no call)
            while (true) {
                try (Socket socket = ss.accept()) {
                    PrintWriter out = new PrintWriter(socket.getOutputStream(), true);
                    out.println("Hello World!");
                }
            }

        }
    }
}

// HelloSSLClient.java

import java.net.*;
import javax.net.*;
import javax.net.ssl.*;
import java.io.*;

public class HelloSSLClient {

    public static void main(String args[]) throws Exception {
        String host = "localhost";
        int port = 1234;
        SocketFactory factory = SSLSocketFactory.getDefault();
        try (Socket connection = factory.createSocket(host, port)) {
            SSLSocket ssl = (SSLSocket) connection;

            SSLParameters sslParams = new SSLParameters();
            sslParams.setEndpointIdentificationAlgorithm("HTTPS");
            ssl.setSSLParameters(sslParams);

            BufferedReader input =
                new BufferedReader(new InputStreamReader(connection.getInputStream()));
            String message = input.readLine();
            System.out.println("Got the message: " + message);
        }
    }
}

(a server's certificate will also be needed in the server's keystore and in the client's truststore for testing any SSL application)

Here is the outcome (when the error occurs):

$ java -Djavax.net.ssl.keyStore=serverkeystore.ks -Djavax.net.ssl.keyStorePassword=xxxxxxx HelloSSLServer

$ java -Djavax.net.ssl.trustStore=clienttruststore.ks -Djavax.net.ssl.trustStorePassword=xxxxxx HelloSSLClient
Exception in thread "main" javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307)
    at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:285)
    at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:180)
    at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164)
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1180)
    at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1091)
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
    at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:721)
    at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:804)
    at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
    at java.base/java.io.BufferedReader.fill(BufferedReader.java:161)
    at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326)
    at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392)
    at HelloSSLClient.main(HelloSSLClient.java:21)

As suggested by @SvetlinZarev and @user207421, here are some snippets of the output with the -Djavax.net.debug=ssl:handshake enabled. Since the code works when one forces the TLSv1.2 protocol, it is more interesting to see what's going on when one simply uses the default (call to ssl.setEnabledProtocols commented out):

NOTE: the project is personal and experimental and the server's certificate is fake, self-signed, and added to the client's truststore (the keystore and the truststore too are dummy), and would be public anyway, so I do not fear to release any sensitive information

From the client side:

java -Djavax.net.debug=ssl:handshake -Djavax.net.ssl.trustStore=clienttruststore.ks -Djavax.net.ssl.trustStorePassword=xxxxxxxxx HelloSSLClient
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.229 CEST|SSLCipher.java:463|jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|WARNING|01|main|2019-08-22 09:54:22.451 CEST|ServerNameExtension.java:261|Unable to indicate server name
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.451 CEST|SSLExtensions.java:257|Ignore, context unavailable extension: server_name
javax.net.ssl|WARNING|01|main|2019-08-22 09:54:22.455 CEST|SignatureScheme.java:282|Signature algorithm, ed25519, is not supported by the underlying providers
javax.net.ssl|WARNING|01|main|2019-08-22 09:54:22.455 CEST|SignatureScheme.java:282|Signature algorithm, ed448, is not supported by the underlying providers
javax.net.ssl|INFO|01|main|2019-08-22 09:54:22.459 CEST|AlpnExtension.java:161|No available application protocols
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.460 CEST|SSLExtensions.java:257|Ignore, context unavailable extension: application_layer_protocol_negotiation
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.460 CEST|SSLExtensions.java:257|Ignore, context unavailable extension: cookie
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.477 CEST|SSLExtensions.java:257|Ignore, context unavailable extension: renegotiation_info
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.477 CEST|PreSharedKeyExtension.java:633|No session to resume.
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.478 CEST|SSLExtensions.java:257|Ignore, context unavailable extension: pre_shared_key
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.482 CEST|ClientHello.java:653|Produced ClientHello handshake message (
"ClientHello": {
  "client version"      : "TLSv1.2",
  "random"              : "8D 95 FA BE D8 F4 BC AC E6 15 36 FE FE A2 57 C4 DD EF F6 53 B8 54 1D 4A ED AE C6 0A CD 92 E0 A4",
  "session id"          : "CC EE 0F 29 F8 9A 3B 72 61 61 99 46 AA 69 CF 23 4F E9 05 13 2A 52 B8 1D 34 18 FA DF 26 1B 46 87",
  "cipher suites"       : "[TLS_AES_128_GCM_SHA256(0x1301), TLS_AES_256_GCM_SHA384(0x1302), TLS_CHACHA20_POLY1305_SHA256(0x1303), TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384(0xC02C), TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256(0xC02B), TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256(0xCCA9), TLS_DHE_DSS_WITH_AES_128_CBC_SHA256(0x0040), 

[...]

TLS_ECDH_RSA_WITH_AES_128_CBC_SHA(0xC00E), TLS_DHE_RSA_WITH_AES_128_CBC_SHA(0x0033), TLS_DHE_DSS_WITH_AES_128_CBC_SHA(0x0032), TLS_EMPTY_RENEGOTIATION_INFO_SCSV(0x00FF)]",

[...]

javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.528 CEST|ServerHello.java:871|Consuming ServerHello handshake message (
"ServerHello": {
  "server version"      : "TLSv1.2",
  "random"              : "B5 27 FE 28 29 85 AC 1A C4 62 57 28 45 12 63 BA 4D CC 4B E0 02 A4 A9 7A ED 9F A3 8D A6 98 85 BE",
  "session id"          : "CC EE 0F 29 F8 9A 3B 72 61 61 99 46 AA 69 CF 23 4F E9 05 13 2A 52 B8 1D 34 18 FA DF 26 1B 46 87",
  "cipher suite"        : "TLS_AES_128_GCM_SHA256(0x1301)",
  "compression methods" : "00",
  "extensions"          : [
    "supported_versions (43)": {
      "selected version": [TLSv1.3]
    },
    "key_share (51)": {
      "server_share": {

[...]

javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.528 CEST|SSLExtensions.java:189|Consumed extension: supported_versions
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.529 CEST|ServerHello.java:967|Negotiated protocol version: TLSv1.3
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.530 CEST|SSLExtensions.java:160|Ignore unsupported extension: server_name
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.530 CEST|SSLExtensions.java:160|Ignore unsupported extension: max_fragment_length
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.530 CEST|SSLExtensions.java:160|Ignore unsupported extension: status_request

[...]

javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.582 CEST|SSLExtensions.java:170|Ignore unavailable extension: server_name
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.582 CEST|SSLExtensions.java:170|Ignore unavailable extension: max_fragment_length
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.583 CEST|SSLExtensions.java:189|Consumed extension: supported_groups
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.583 CEST|SSLExtensions.java:204|Ignore unavailable extension: server_name
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.583 CEST|SSLExtensions.java:204|Ignore unavailable extension: max_fragment_length
javax.net.ssl|WARNING|01|main|2019-08-22 09:54:22.583 CEST|SSLExtensions.java:212|Ignore impact of unsupported extension: supported_groups
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.583 CEST|SSLExtensions.java:204|Ignore unavailable extension: application_layer_protocol_negotiation
javax.net.ssl|DEBUG|01|main|2019-08-22 09:54:22.585 CEST|Alert.java:238|Received alert message (
"Alert": {
  "level"      : "fatal",
  "description": "handshake_failure"
}
)
javax.net.ssl|ERROR|01|main|2019-08-22 09:54:22.586 CEST|TransportContext.java:312|Fatal (HANDSHAKE_FAILURE): Received fatal alert: handshake_failure (
"throwable" : {
  javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)

And here is the server side:

java -Djavax.net.debug=ssl,handshake -Djavax.net.ssl.keyStore=serverkeystore.ks -Djavax.net.ssl.keyStorePassword=xxxxxxxxx HelloSSLServer
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:55.869 CEST|SSLCipher.java:463|jdk.tls.keyLimits:  entry = AES/GCM/NoPadding KeyUpdate 2^37. AES/GCM/NOPADDING:KEYUPDATE = 137438953472
javax.net.ssl|WARNING|01|main|2019-08-22 12:23:59.697 CEST|SignatureScheme.java:282|Signature algorithm, ed25519, is not supported by the underlying providers
javax.net.ssl|WARNING|01|main|2019-08-22 12:23:59.698 CEST|SignatureScheme.java:282|Signature algorithm, ed448, is not supported by the underlying providers
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.704 CEST|ClientHello.java:809|Consuming ClientHello handshake message (
"ClientHello": {
  "client version"      : "TLSv1.2",
  "random"              : "0D E6 53 8A B0 E4 E7 9A 80 93 49 84 AD 88 0A 5F D5 7F 29 37 C3 86 A7 28 A7 D9 C6 7D EB DF 6A 3D",
  "session id"          : "5B 9A 18 25 31 65 8C 8F E8 E6 93 DA F5 AA 50 45 A8 C9 20 D1 9D 67 35 9B 7B D3 46 D5 CA C0 FC 85",
  "cipher suites"       : "[TLS_AES_128_GCM_SHA256(0x1301), TLS_AES_256_GCM_SHA384(0x1302), TLS_CHACHA20_POLY1305_SHA256(0x1303), TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384(0xC02C), TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256(0xC02B),

[...]

TLS_DHE_DSS_WITH_AES_128_CBC_SHA(0x0032), TLS_EMPTY_RENEGOTIATION_INFO_SCSV(0x00FF)]",
  "compression methods" : "00",
  "extensions"          : [
    "status_request (5)": {
      "certificate status type": ocsp
      "OCSP status request": {
        "responder_id": <empty>
        "request extensions": {
          <empty>
        }
      }
    },
    "supported_groups (10)": {
      "versions": [secp256r1, secp384r1, secp521r1, sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1, secp256k1, ffdhe2048, ffdhe3072, ffdhe4096, ffdhe6144, ffdhe8192]
    },
    "ec_point_formats (11)": {
      "formats": [uncompressed]
    },
    "signature_algorithms (13)": {
      "signature schemes": [ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_secp521r1_sha512, rsa_pss_rsae_sha256, rsa_pss_rsae_sha384, rsa_pss_rsae_sha512, rsa_pss_pss_sha256, rsa_pss_pss_sha384, rsa_pss_pss_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha512, dsa_sha256, ecdsa_sha224, rsa_sha224, dsa_sha224, ecdsa_sha1, rsa_pkcs1_sha1, dsa_sha1]
    },
    "signature_algorithms_cert (50)": {
      "signature schemes": [ecdsa_secp256r1_sha256, ecdsa_secp384r1_sha384, ecdsa_secp521r1_sha512, rsa_pss_rsae_sha256, rsa_pss_rsae_sha384, rsa_pss_rsae_sha512, rsa_pss_pss_sha256, rsa_pss_pss_sha384, rsa_pss_pss_sha512, rsa_pkcs1_sha256, rsa_pkcs1_sha384, rsa_pkcs1_sha512, dsa_sha256, ecdsa_sha224, rsa_sha224, dsa_sha224, ecdsa_sha1, rsa_pkcs1_sha1, dsa_sha1]
    },

[...]

javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.704 CEST|SSLExtensions.java:189|Consumed extension: supported_versions
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.705 CEST|ClientHello.java:839|Negotiated protocol version: TLSv1.3
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.705 CEST|SSLExtensions.java:189|Consumed extension: psk_key_exchange_modes
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.706 CEST|PreSharedKeyExtension.java:805|Handling pre_shared_key absence.
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.706 CEST|SSLExtensions.java:170|Ignore unavailable extension: server_name
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.706 CEST|SSLExtensions.java:170|Ignore unavailable extension: max_fragment_length
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.706 CEST|SSLExtensions.java:189|Consumed extension: status_request
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.707 CEST|SSLExtensions.java:189|Consumed extension: supported_groups
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.707 CEST|SSLExtensions.java:160|Ignore unsupported extension: ec_point_formats
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.707 CEST|SSLExtensions.java:189|Consumed extension: signature_algorithms
javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.707 CEST|SSLExtensions.java:189|Consumed extension: signature_algorithms_cert

[...]

javax.net.ssl|DEBUG|01|main|2019-08-22 12:23:59.739 CEST|ServerHello.java:576|Produced ServerHello handshake message (
"ServerHello": {
  "server version"      : "TLSv1.2",
  "random"              : "FD FD 39 0C 3A D8 F0 E8 38 F8 08 D8 19 94 7A FA 4A 68 71 F8 4C 32 EB 7A D0 53 96 E5 9F E8 0A 3B",
  "session id"          : "5B 9A 18 25 31 65 8C 8F E8 E6 93 DA F5 AA 50 45 A8 C9 20 D1 9D 67 35 9B 7B D3 46 D5 CA C0 FC 85",
  "cipher suite"        : "TLS_AES_128_GCM_SHA256(0x1301)",
  "compression methods" : "00",
  "extensions"          : [
    "supported_versions (43)": {
      "selected version": [TLSv1.3]
    },
    "key_share (51)": {
      "server_share": {

[...]

javax.net.ssl|ALL|01|main|2019-08-22 12:23:59.785 CEST|X509Authentication.java:243|No X.509 cert selected for EC
javax.net.ssl|WARNING|01|main|2019-08-22 12:23:59.785 CEST|CertificateMessage.java:1055|Unavailable authentication scheme: ecdsa_sha1
javax.net.ssl|ALL|01|main|2019-08-22 12:23:59.785 CEST|X509Authentication.java:243|No X.509 cert selected for RSA
javax.net.ssl|WARNING|01|main|2019-08-22 12:23:59.785 CEST|CertificateMessage.java:1055|Unavailable authentication scheme: rsa_pkcs1_sha1
javax.net.ssl|WARNING|01|main|2019-08-22 12:23:59.785 CEST|CertificateMessage.java:1065|No available authentication scheme
javax.net.ssl|ERROR|01|main|2019-08-22 12:23:59.788 CEST|TransportContext.java:312|Fatal (HANDSHAKE_FAILURE): No available authentication scheme (
"throwable" : {
  javax.net.ssl.SSLHandshakeException: No available authentication scheme
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)

[...]
like image 926
Corentor Avatar asked Aug 22 '19 02:08

Corentor

People also ask

How do I fix SSL handshake failure in Java?

In order to solve this error, just add the root certificate required to connect to the server in question into the truststore of your application's JRE. If you don't know, JVM uses two files, keystore and truststore to store the SSL certificates.

What causes SSLHandshakeException?

The SSLHandshakeException is thrown when an error occurs while a client and server connection fails to agree on their desired security level. This exception is one of a handful of classes that inherits from the parent SSLException class.

What does Handshake_failure mean?

A TLS/SSL handshake failure occurs when a client and server cannot establish communication using the TLS/SSL protocol. When this error occurs in Apigee Edge, the client application receives an HTTP status 503 with the message Service Unavailable.

1 Answers

Here is the solution: one needs the -keyalg flag with keytool to generate certificates, otherwise, the key will be ciphered with the old default DSA, that is not allowed anymore with TLS1.3. With RSA it works.

Conclusion:

  • keytool ciphers by default with DSA for backwards compatibility, unless -keyalg is provided
  • Since Java 11, TLS1.3 is the new default encryption scheme for SSL sockets in JSSE, when it can be negotiated.
  • DSA is not supported anymore in TLS1.3

It is therefore recommended to always generate certificates with explicit -keyalg RSA (or any other supported TLS1.3 algorithm) to avoid surprises.

Edit 19 nov 2020: the issue of the keytool's default algorithm was discussed on Red Hat's Bugzilla a long time ago ( https://bugzilla.redhat.com/show_bug.cgi?id=1582253 ). It seems that RSA is the keytool's new default algorithm in OpenJDK 14, but it is not guaranteed for other JDK implementations to my knowledge (unless the JDK specification says anything about it).

like image 102
Corentor Avatar answered Sep 19 '22 17:09

Corentor
Sign in to Comment

Related questions

Rescuing a swallowed Exception in Java
Chaining lambda functions [duplicate]
length of array of private class is not accessible
Apache Commons CLI 1.3.1: How to ignore unknown Arguments?
Android - custom button with shape drawables and a gradient programmatically
custom soapfault resolver triggering but not finding SoapFault in SoapBody
@TestPropertySource annotation alternative on a method level
PyCharm - Unable to save settings - Failed to save settings. Please restart PyCharm Community Edition
Error:No toolchains found in the NDK toolchains folder for ABI with prefix: arm-linux-androideabi
Java audio visualizer- How to capture real-time sound output to speaker?
kafka consumer polling timeout
Custom Collapsing Toolbar smooth title resizing
How to say Hystrix not to trigger fallback for some of the exceptions in Hystrix command
How do I bundle a JRE in my JAR, so that it can run on systems without Java?
Android studio ; jni.h not found
Serialization and deserialization of lambda
Encryption and Decryption between Java and Javascript won't work
Gradle build failed with message: TaskDependencyResolveException: Could not determine the dependencies of task ':compileDebugKotlin'
Add ssl certificate to selenium-webdriver
Why is this Java method call considered ambiguous?

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!