Changes in SSLEngine usage when going up to TLSv1.3

Question

Java 11 was released with TLSv1.3 support, used by default.

It works OK in context of HTTPS and SSL sockets, but it seems that when using SSLEngine there are additional hurdles due to changes in TLSv1.3 behavior.

So there is a robust implementation of communication via NIO using SSLEngine that no longer works when TLSv1.3 is enabled. There are no obvious errors, in form of exceptions or SSL errors, two nodes will just send wrap/unwrap messages back and forth and eventually timeout.

I am interested in an exact list of behavior changes between SSLEngine using TLSv1.2 and SSLEngine using TLSv1.3, and if possible a migration checklist between these. Unfortunately, SSLEngine javadocs of Java 11 does not have this information - no new methods in Java 11 and no reference to TLSv1.3.

Answer 1

It's true that there is no explicit reference to the impact of TLS 1.3 on SSLEngine in its Javadoc in JDK 11, and there were no changes in its methods.

However the fifth item (closure) in the list of phases of SSLEngine was updated in the general description at the start of its Javadoc in JDK 11:

Closure - When the connection is no longer needed, the client and the server applications should each close both sides of their respective connections. For SSLEngine objects, an application should call closeOutbound() and send any remaining messages to the peer. Likewise, an application should receive any remaining messages from the peer before calling closeInbound(). The underlying transport mechanism can then be closed after both sides of the SSLEngine have been closed. If the connection is not closed in an orderly manner (for example closeInbound() is called before the peer's write closure notification has been received), exceptions will be raised to indicate that an error has occurred. Once an engine is closed, it is not reusable: a new SSLEngine must be created.

The change is also discussed in Oracle's Release Notes for JDK 11:

TLS 1.3 Half-Close Policy
A new system property, jdk.tls.acknowledgeCloseNotify, has been added. The default value of the system property is false. If the system property is set to true, a corresponding close_notify alert will be sent when receiving a close_notify alert, and the connection will be duplex closed.

TLS 1.2 and prior versions use a duplex-close policy, while TLS 1.3 uses a half-close policy. The inbound and the outbound close_notify alerts for TLS 1.3 are independent. When upgrading to TLS 1.3, unexpected behavior can occur if your application shuts down the (D)TLS connection by using only one of the SSLEngine.closeInbound() or SSLEngine.closeOutbound() APIs, but not both in each side of the connection. If your application exhibits unexpected hangs or timeouts when the underlying (D)TLS transportation is not duplex closed, you may need to set this property to true.

Note that when a TLS/DTLS connection is no longer needed, the client and server applications should each close both sides of their respective connection.

So setting jdk.tls.acknowledgeCloseNotify to true might resolve your specific concern about timeouts when using SSlEngine with TLS 1.3:

If your application exhibits unexpected hangs or timeouts when the underlying (D)TLS transportation is not duplex closed, you may need to set this property to true.

The Release Notes document also links to the closed JDK bug JDK-8208526 : TLS 1.3 half-close and synchronization issues which discusses the change in even greater detail.

The related (and closed) bug JDK-8207009 : TLS 1.3 half-close and synchronization issues may also be of interest.

Other References:

• See "Appendix D. Backward Compatibility" of RFC 8446: "The Transport Layer Security (TLS) Protocol Version 1.3" (pp. 138-141).
• There's a brief discussion of compatibility between TLS 1.3 and earlier releases in this Oracle video "Monday Technical Sessions: Moscone West 2004" between 2:53:37 and 2:56:35.