SSLHandshakeException: Received fatal alert: handshake_failure when setting ciphers on tomcat 7 server

Question

I have a Tomcat7 web-server which I tried to configure to accept secure connections by adding this connector to the server.xml file:

<Connector SSLEnabled="true"
           acceptCount="100"
           connectionTimeout="20000"
           executor="tomcatThreadPool"
           keyAlias="server"
           keystoreFile="c:\opt\engine\conf\tc.keystore"
           keystorePass="o39UI12z"
           maxKeepAliveRequests="15"
           port="8443"
           protocol="HTTP/1.1"
           redirectPort="8443"
           scheme="https"
           secure="true"
           sslProtocol="TLS"/>

I'm using a self-signed certificate generated using this command:

%JAVA_HOME%/bin/keytool -genkeypair -keystore c:\opt\engine\conf\tc.keystore -storepass o39UI12z-keypass o39UI12z-dname "cn=Company, ou=Company, o=Com, c=US" -alias server -validity 36500

On the client side I have a spring application that connects with the server using RestTemplate. On application context startup I initalize the restTemplate instance this way:

final ClientHttpRequestFactory clientHttpRequestFactory =
        new MyCustomClientHttpRequestFactory(new NullHostNameVerifier(), serverInfo);
    restTemplate.setRequestFactory(clientHttpRequestFactory);

The class MyCustomClientHttpRequestFactory looks like this:

public class MyCustomClientHttpRequestFactory  extends SimpleClientHttpRequestFactory {

private static final Logger LOGGER = LoggerFactory
    .getLogger(MyCustomClientHttpRequestFactory.class);

private final HostnameVerifier hostNameVerifier;
private final ServerInfo serverInfo;

public MyCustomClientHttpRequestFactory (final HostnameVerifier hostNameVerifier,
    final ServerInfo serverInfo) {
    this.hostNameVerifier = hostNameVerifier;
    this.serverInfo = serverInfo;
}

@Override
protected void prepareConnection(final HttpURLConnection connection, final String httpMethod)
    throws IOException {
    if (connection instanceof HttpsURLConnection) {
        ((HttpsURLConnection) connection).setHostnameVerifier(hostNameVerifier);
        ((HttpsURLConnection) connection).setSSLSocketFactory(initSSLContext()
            .getSocketFactory());
    }
    super.prepareConnection(connection, httpMethod);
}

private SSLContext initSSLContext() {
    try {
        System.setProperty("https.protocols", "TLSv1");

        // Set ssl trust manager. Verify against our server thumbprint
        final SSLContext ctx = SSLContext.getInstance("TLSv1");
        final SslThumbprintVerifier verifier = new SslThumbprintVerifier(serverInfo);
        final ThumbprintTrustManager thumbPrintTrustManager =
            new ThumbprintTrustManager(null, verifier);
        ctx.init(null, new TrustManager[] { thumbPrintTrustManager }, null);
        return ctx;
    } catch (final Exception ex) {
        LOGGER.error(
            "An exception was thrown while trying to initialize HTTP security manager.", ex);
        return null;
    }
}

Up until here everything worked fine. When I put a break point in the SslThumbprintVerifier class, the code reached to this point and also to the NullHostNameVerifier class. This was tested in production and worked great.

Now I wanted to extend security by limiting the cipher suites and I added this property to the Connector I presented at the beginning:

ciphers="TLS_KRB5_WITH_RC4_128_SHA,SSL_RSA_WITH_RC4_128_SHA,SSL_DHE_RSA_WITH_AES_256_CBC_SHA,SSL_RSA_WITH_AES_256_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_KRB5_WITH_3DES_EDE_CBC_SHA"

Now when I'm running the client code I get this exception:

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

This happens in the restTemplate.doExecute() method. The code doesn't reach to the thumbprint verifier class nor to the host name verifier class as it did before adding the ciphers.

In debug I checked the ctx.getSocketFactory().getSupportedCipherSuites() which showed:

[SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, SSL_RSA_WITH_NULL_MD5, SSL_RSA_WITH_NULL_SHA, SSL_DH_anon_WITH_RC4_128_MD5, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_256_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_DH_anon_EXPORT_WITH_RC4_40_MD5, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_KRB5_WITH_RC4_128_SHA, TLS_KRB5_WITH_RC4_128_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_RC4_40_SHA, TLS_KRB5_EXPORT_WITH_RC4_40_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5]

and also checked ctx.getSocketFactory().getDefaultCipherSuites():

[SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]

and lastly the ctx.getSupportedSSLParameters().getCipherSuites():

[SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, SSL_RSA_WITH_NULL_MD5, SSL_RSA_WITH_NULL_SHA, SSL_DH_anon_WITH_RC4_128_MD5, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_256_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_DH_anon_EXPORT_WITH_RC4_40_MD5, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_KRB5_WITH_RC4_128_SHA, TLS_KRB5_WITH_RC4_128_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_RC4_40_SHA, TLS_KRB5_EXPORT_WITH_RC4_40_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5]

As far as I understand, if there's an intersection between the client's supported/enabled cipher suites and the server's supported cipher suites it should work (we have such intersection with SSL_RSA_WITH_RC4_128_SHA for example). And yet I'm getting this error.

In the next step I added this java parameter:

-Djavax.net.debug=ssl,handshake,failure

The log showed only the client-hello, with no server-hello response:

[2013-03-20 15:29:51.315] [INFO ] data-service-pool-37          System.out                                                        trigger seeding of SecureRandom 
[2013-03-20 15:29:51.315] [INFO ] data-service-pool-37          System.out                                                        done seeding SecureRandom 
[2013-03-20 15:30:38.894] [INFO ] data-service-pool-37          System.out                                                        Allow unsafe renegotiation: false 
[2013-03-20 15:30:38.894] [INFO ] data-service-pool-37          System.out                                                        Allow legacy hello messages: true 
[2013-03-20 15:30:38.894] [INFO ] data-service-pool-37          System.out                                                        Is initial handshake: true 
[2013-03-20 15:30:38.894] [INFO ] data-service-pool-37          System.out                                                        Is secure renegotiation: false 
[2013-03-20 15:30:38.894] [INFO ] data-service-pool-37          System.out                                                        %% No cached client session 
[2013-03-20 15:30:38.894] [INFO ] data-service-pool-37          System.out                                                        *** ClientHello, TLSv1 
[2013-03-20 15:30:38.941] [INFO ] data-service-pool-37          System.out                                                        RandomCookie:  GMT: 1363720446 bytes = { 99, 249, 173, 214, 110, 82, 58, 52, 189, 92, 74, 169, 133, 128, 250, 109, 160, 64, 112, 253, 50, 160, 255, 196, 85, 93, 33, 172 } 
[2013-03-20 15:30:38.941] [INFO ] data-service-pool-37          System.out                                                        Session ID:  {} 
[2013-03-20 15:30:38.941] [INFO ] data-service-pool-37          System.out                                                        Cipher Suites: [SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV] 
[2013-03-20 15:30:38.956] [INFO ] data-service-pool-37          System.out                                                        Compression Methods:  { 0 } 
[2013-03-20 15:30:38.956] [INFO ] data-service-pool-37          System.out                                                        *** 
[2013-03-20 15:30:38.956] [INFO ] data-service-pool-37          System.out                                                        data-service-pool-37, WRITE: TLSv1 Handshake, length = 81 
[2013-03-20 15:30:38.956] [INFO ] data-service-pool-37          System.out                                                        data-service-pool-37, READ: TLSv1 Alert, length = 2 
[2013-03-20 15:30:38.956] [INFO ] data-service-pool-37          System.out                                                        data-service-pool-37, RECV TLSv1 ALERT:  fatal, handshake_failure 
[2013-03-20 15:30:38.972] [INFO ] data-service-pool-37          System.out                                                        data-service-pool-37, called closeSocket() 
[2013-03-20 15:30:38.972] [INFO ] data-service-pool-37          System.out                                                        data-service-pool-37, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure 

When removing the ciphers from the server.xml it works again. Also I will mention that I'm testing both the server and the client on the same machine. I tried setting the server's ip in the client request to 'localhost' and to the actual machine ip and it didn't work in both cases. Also, I had this server running on a different linux machine (the keystore was generated on the linux machine with a linux path of course) and still - it works without the ciphers and stops working with the ciphers.

Answer 1

Well, I got this issue solved. It appears that by creating a self-signed certificate, using keytool, without providing -keyalg parameter makes the key-pair algorithm default to DSA. None of my ciphers suite included DSA algorithm. In that case, although the client and the server had intersection between their cipher-suites, neither was suitable for the key algoritm.

Adding -keyalg RSA when generating the keystore, solved the problem.