Android SSLSocket handshake failure in Android 6 and above

Question

I've written a server based on a Java SSLServerSocket that accepts connections and communicates to android applications via a custom binary protocol:

ServerSocket serverSocket = SSLServerSocketFactory.getDefault().createServerSocket(1234);
while (true) {
    Socket socket = serverSocket.accept();
    ...
}

I run the server with the following arguments:

-Djavax.net.ssl.keyStore=keystore.jks
-Djavax.net.ssl.keyStorePassword=<PASSWORD>

And the certificate is generated using the following tutorial which builds a public and private key set: http://judebert.com/progress/archives/425-Using-SSL-in-Java,-Part-2.html:

keytool -genkeypair -keystore keystore.jks -alias keyname
keytool -export -alias keyname -file keyname.crt -keystore keystore.jks 
keytool -importcert -file keyname.crt -keystore truststore.jks

Also, I make this compatible with android by building a truststore using bouncycastle:

keytool -importkeystore -srckeystore truststore.jks -srcstoretype JKS -srcstorepass <PASSWORD> -destkeystore truststore.bks -deststoretype BKS -deststorepass <PASSWORD> -provider org.bouncycastle.jce.provider.BouncyCastleProvider -providerpath bcprov-ext-jdk15on-1.58.jar

Download the bouncycastle provider here: https://www.bouncycastle.org/latest_releases.html

And moved the resulting truststore.bks into the raw resource folder.

On Android I use the following code to build a SSLSocketFactory which allows me to import the generated bouncycastle certificate which authenticates me against the server:

KeyStore trustStore = KeyStore.getInstance("BKS");
InputStream trustStoreStream = context.getResources().openRawResource(R.raw.truststore);
trustStore.load(trustStoreStream, "<PASSWORD>".toCharArray());

TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(trustStore);

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, trustManagerFactory.getTrustManagers(), null);

Socket socket = sslContext.getSocketFactory().createSocket("ip", 1234);
... use socket

This works well for Android versions below 6. My issue is at version 6 and higher I get an exception when trying to use the socket:

 Shutting down connection Socket[address=/ip,port=1234,localPort=321321] due to exception Handshake failed
 javax.net.ssl.SSLHandshakeException: Handshake failed
    at com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:429)
    at com.example.Client.connect(Client.java:97)
    at com.example.Client.start(Client.java:60)
    at com.example.BackendServiceFactory$2.call(BackendServiceFactory.java:136)
    at com.example.BackendServiceFactory$2.call(BackendServiceFactory.java:130)
    ...
 Caused by: javax.net.ssl.SSLProtocolException: SSL handshake terminated: ssl=0xe69ec900: Failure in SSL library, usually a protocol error
 error:10000410:SSL routines:OPENSSL_internal:SSLV3_ALERT_HANDSHAKE_FAILURE (external/boringssl/src/ssl/s3_pkt.c:641 0xe2d10880:0x00000001)
 error:1000009a:SSL routines:OPENSSL_internal:HANDSHAKE_FAILURE_ON_CLIENT_HELLO (external/boringssl/src/ssl/s3_clnt.c:800 0xe6ea5af3:0x00000000)
    at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method)
    at com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:357)
    ... 24 more

I'm not sure whats going on here. There seems to be a misstep along the way dealing with client certificates, could this be a mismatch of cipher suites?

I've put together a minimal example with a Java server, Java client and Android client to help diagnose this issue here:

https://github.com/johncarl81/androidCA

Answer 1

I figured this was going to be a simple fix. Seems that I needed to specify the key algorithm in the very first keytool command:

keytool -genkeypair -keystore keystore.jks -alias keyname -keyalg RSA

This generates a 2048 bit RSA key which is compatible with versions of android < 6 and >= 6.