Java SSL: "fatal error: 80: ... unwrapping net record" after adding the HTTPS EndpointIdentificationAlgorithm

Question

Java 7u9

Error msg in title is: "fatal error: 80: problem unwrapping net record". SO wouldn't let me put "problem" in the title.

I am building a Java HTTPS client against Netty. The SSL handshake was working until I added added the "HTTPS" endpoint identification algorithm to enable server hostname verification:

SSLEngine engine = tcpHelper.getSSLContext().createSSLEngine();
SSLParameters sslParameters = engine.getSSLParameters();
sslParameters.setEndpointIdentificationAlgorithm("HTTPS");
engine.setSSLParameters(sslParameters);
engine.setUseClientMode(true);

After adding the algorithm, the SSL handshake hangs and the connection eventually times out. With SSL debugging enabled (javax.net.debug=all), I can see that the handshake now fails after ServerHello, after the server sends it's cert chain, after

*** ServerHelloDone
1761586552@qtp-1653588482-2, WRITE: TLSv1.2 Handshake, length = 3294

on the server. The client receives and displays the cert chain, and then fails with:

New I/O  worker #3, fatal error: 80: problem unwrapping net record
java.lang.RuntimeException: Delegated task threw Exception/Error
%% Invalidated:  [Session-1, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256]
New I/O  worker #3, SEND TLSv1.2 ALERT:  fatal, description = internal_error
New I/O  worker #3, WRITE: TLSv1.2 Alert, length = 2

Answer 1

Java 7u9

First, thanks to Bruno for his help on this related question which lead me to final answer.

Answering my own question. Hard won knowledge.

The solution is to add the host and port of the request target to the constructor when you create the SSL Engine:

SSLEngine engine = tcpHelper.getSSLContext().createSSLEngine(targetHost, targetPort);

Without this, Java will ultimately throw a NullPointerException way down deep in the SSL libs (IPAddressUtil.textToNumericFormatV4), which results in the not-very-helpful error message in the SSL debug output.