Netty SslHandler headache

Question

I've wasted a couple of days now trying to track down an intermittent bug in the newly added Akka transport encryption.

NOTE: I have experimented with setting setIssueHandshake(true) on either or both of the server and the clients, but it doesn't help at all.

Our cipher spec tests a couple of different ciphers from different suites, to make sure that the settings we support actually works. However, the tests can sometimes pass 10 times, and then start failing every other test, it's truly SecureRandomly failing ;-) Please do note that the test fails even on SHA1PRNG so it is clearly unrelated to the additional ciphers we provide.

The code that creates the SslHandler: https://github.com/akka/akka/blob/wip-ssl-unbroken-%E2%88%9A/akka-remote/src/main/scala/akka/remote/netty/NettySSLSupport.scala

The code that constructs the pipeline: https://github.com/akka/akka/blob/wip-ssl-unbroken-%E2%88%9A/akka-remote/src/main/scala/akka/remote/netty/NettyRemoteSupport.scala#L66

The tests: https://github.com/akka/akka/blob/wip-ssl-unbroken-%E2%88%9A/akka-remote/src/test/scala/akka/remote/Ticket1978CommunicationSpec.scala

The fall-back config (for what the test above doesn't override): https://github.com/akka/akka/blob/wip-ssl-unbroken-%E2%88%9A/akka-remote/src/main/resources/reference.conf

The keystore & truststore to use for the test: https://github.com/akka/akka/tree/wip-ssl-unbroken-%E2%88%9A/akka-remote/src/test/resources

The root exception that fails the test is:

**java.security.InvalidKeyException: No installed provider supports this key: (null)**
    at javax.crypto.Cipher.a(DashoA13*..)
    at javax.crypto.Cipher.init(DashoA13*..)
    at javax.crypto.Cipher.init(DashoA13*..)
    at com.sun.net.ssl.internal.ssl.CipherBox.<init>(CipherBox.java:88)
    at com.sun.net.ssl.internal.ssl.CipherBox.newCipherBox(CipherBox.java:119)
    at com.sun.net.ssl.internal.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:369)
    at com.sun.net.ssl.internal.ssl.Handshaker.newReadCipher(Handshaker.java:410)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.changeReadCiphers(SSLEngineImpl.java:550)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1051)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:845)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
    at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:969)
    at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:670)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:333)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:91)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:373)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:247)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)

And the "full" one is:

[ERROR] [06/20/2012 10:38:33.670] [remote-sys-4] [ActorSystem(remote-sys)] RemoteServerError@akka://remote-sys@localhost:59104] Error[
javax.net.ssl.SSLException: Algorithm missing:  
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.changeReadCiphers(SSLEngineImpl.java:554)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1051)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:845)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:721)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:607)
    at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:969)
    at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:670)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:333)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:91)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:373)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:247)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)
Caused by: java.security.NoSuchAlgorithmException: Could not create cipher AES/128
    at com.sun.net.ssl.internal.ssl.CipherBox.<init>(CipherBox.java:99)
    at com.sun.net.ssl.internal.ssl.CipherBox.newCipherBox(CipherBox.java:119)
    at com.sun.net.ssl.internal.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:369)
    at com.sun.net.ssl.internal.ssl.Handshaker.newReadCipher(Handshaker.java:410)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.changeReadCiphers(SSLEngineImpl.java:550)
    ... 17 more
Caused by: java.security.InvalidKeyException: No installed provider supports this key: (null)
    at javax.crypto.Cipher.a(DashoA13*..)
    at javax.crypto.Cipher.init(DashoA13*..)
    at javax.crypto.Cipher.init(DashoA13*..)
    at com.sun.net.ssl.internal.ssl.CipherBox.<init>(CipherBox.java:88)
    ... 21 more
]

Answer 1

Not a bug in Netty, there was an unfortunate write-race between the application-level handshake and the SSL handshake. Worth to note is that setIssueHandshake(true) doesn't seem to be transparently handling the handshake as you need to manually back off writes until the handshake is completed.

Answer 2

whilst I've not seen exactly this exception, it's certainly the case that a javax.crypto.Cipher is not thread safe; I have an application where I finally tracked down a bug which was solved by synchronizing on the cipher:

cipher synchronized { cipher doFinal encryptedBytes }

Apologies if this is not the solution, but you posted a lot of code! (It's probably not exactly the same as the stack trace indicates that the issue is even getting a Cipher instance - but could this also need synchronizing?)