SSL handshake alert: unrecognized_name error since upgrade to Java 1.7.0

Question

Java 7 introduced SNI support which is enabled by default. I have found out that certain misconfigured servers send an "Unrecognized Name" warning in the SSL handshake which is ignored by most clients... except for Java. As @Bob Kerns mentioned, the Oracle engineers refuse to "fix" this bug/feature.

As workaround, they suggest to set the jsse.enableSNIExtension property. To allow your programs to work without re-compiling, run your app as:

java -Djsse.enableSNIExtension=false yourClass

The property can also be set in the Java code, but it must be set before any SSL actions. Once the SSL library has loaded, you can change the property, but it won't have any effect on the SNI status. To disable SNI on runtime (with the aforementioned limitations), use:

System.setProperty("jsse.enableSNIExtension", "false");

The disadvantage of setting this flag is that SNI is disabled everywhere in the application. In order to make use of SNI and still support misconfigured servers:

1. Create a SSLSocket with the host name you want to connect to. Let's name this sslsock.
2. Try to run sslsock.startHandshake(). This will block until it is done or throw an exception on error. Whenever an error occurred in startHandshake(), get the exception message. If it equals to handshake alert: unrecognized_name, then you have found a misconfigured server.
3. When you have received the unrecognized_name warning (fatal in Java), retry opening a SSLSocket, but this time without a host name. This effectively disables SNI (after all, the SNI extension is about adding a host name to the ClientHello message).

For the Webscarab SSL proxy, this commit implements the fall-back setup.

I had what I believe the same issue is. I found that I needed to adjust the Apache configuration to include a ServerName or ServerAlias for the host.

This code failed:

public class a {
   public static void main(String [] a) throws Exception {
      java.net.URLConnection c = new java.net.URL("https://mydomain.com/").openConnection();
      c.setDoOutput(true);
      c.getOutputStream();
   }
}

And this code worked:

public class a {
   public static void main(String [] a) throws Exception {
      java.net.URLConnection c = new java.net.URL("https://google.com/").openConnection();
      c.setDoOutput(true);
      c.getOutputStream();
   }
}

Wireshark revealed that during the TSL/SSL Hello the warning Alert (Level: Warning, Description: Unrecognized Name), Server Hello Was being sent from the server to the client. It was only a warning, however, Java 7.1 then responded immediately back with a "Fatal, Description: Unexpected Message", which I assume means the Java SSL libraries don't like to see the warning of unrecognized name.

From the Wiki on Transport Layer Security (TLS):

112 Unrecognized name warning TLS only; client's Server Name Indicator specified a hostname not supported by the server

This led me to look at my Apache config files and I found that if I added a ServerName or ServerAlias for the name sent from the client/java side, it worked correctly without any errors.

<VirtualHost mydomain.com:443>
  ServerName mydomain.com
  ServerAlias www.mydomain.com

You can disable sending SNI records with the System property jsse.enableSNIExtension=false.

If you can change the code it helps to use SSLCocketFactory#createSocket() (with no host parameter or with a connected socket). In this case it will not send a server_name indication.

We also ran into this error on a new Apache server build.

The fix in our case was to define a ServerAlias in the httpd.conf that corresponded to the host name that Java was trying to connect to. Our ServerName was set to the internal host name. Our SSL cert was using the external host name, but that was not sufficient to avoid the warning.

To help debug, you can use this ssl command:

openssl s_client -servername <hostname> -connect <hostname>:443 -state

If there is a problem with that hostname, then it will print this message near the top of the output:

SSL3 alert read: warning:unrecognized name

I should also note that we did not get that error when using that command to connect to the internal host name, even though it did not match the SSL cert.

Instead of relying on the default virtual host mechanism in apache, you can define one last catchall virtualhost that uses an arbitrary ServerName and a wildcard ServerAlias, e.g.

ServerName catchall.mydomain.com
ServerAlias *.mydomain.com

In that way you can use SNI and apache will not send back the SSL warning.

Of course, this only works if you can describe all of your domains easily using a wildcard syntax.

It should be useful. To retry on a SNI error in Apache HttpClient 4.4 - the easiest way we came up with (see HTTPCLIENT-1522):

public class SniHttpClientConnectionOperator extends DefaultHttpClientConnectionOperator {

    public SniHttpClientConnectionOperator(Lookup<ConnectionSocketFactory> socketFactoryRegistry) {
        super(socketFactoryRegistry, null, null);
    }

    @Override
    public void connect(
            final ManagedHttpClientConnection conn,
            final HttpHost host,
            final InetSocketAddress localAddress,
            final int connectTimeout,
            final SocketConfig socketConfig,
            final HttpContext context) throws IOException {
        try {
            super.connect(conn, host, localAddress, connectTimeout, socketConfig, context);
        } catch (SSLProtocolException e) {
            Boolean enableSniValue = (Boolean) context.getAttribute(SniSSLSocketFactory.ENABLE_SNI);
            boolean enableSni = enableSniValue == null || enableSniValue;
            if (enableSni && e.getMessage() != null && e.getMessage().equals("handshake alert:  unrecognized_name")) {
                TimesLoggers.httpworker.warn("Server received saw wrong SNI host, retrying without SNI");
                context.setAttribute(SniSSLSocketFactory.ENABLE_SNI, false);
                super.connect(conn, host, localAddress, connectTimeout, socketConfig, context);
            } else {
                throw e;
            }
        }
    }
}

and

public class SniSSLSocketFactory extends SSLConnectionSocketFactory {

    public static final String ENABLE_SNI = "__enable_sni__";

    /*
     * Implement any constructor you need for your particular application -
     * SSLConnectionSocketFactory has many variants
     */
    public SniSSLSocketFactory(final SSLContext sslContext, final HostnameVerifier verifier) {
        super(sslContext, verifier);
    }

    @Override
    public Socket createLayeredSocket(
            final Socket socket,
            final String target,
            final int port,
            final HttpContext context) throws IOException {
        Boolean enableSniValue = (Boolean) context.getAttribute(ENABLE_SNI);
        boolean enableSni = enableSniValue == null || enableSniValue;
        return super.createLayeredSocket(socket, enableSni ? target : "", port, context);
    }
}

and

cm = new PoolingHttpClientConnectionManager(new SniHttpClientConnectionOperator(socketFactoryRegistry), null, -1, TimeUnit.MILLISECONDS);