OpenJDK 11 problem - Client finished handshake before last UNWRAP

Question

I am migrating our code base from Oracle Java 1.8.0_131 to OpenJDK 11.0.1. We have code that implements nio-ssl socket channels. In Java 8, the client/server handshake works fine. In Java 11, the client finishes the handshake before it unwraps the last handshake message from the server.

To produce this problem, I am just establishing a connection between the client and server and letting them perform a SSL handshake. I am NOT sending any additional data through.

I establish the connection using java 8 and get the output below. I then compile, build and run the same code using java 11 and get the other output below. I do not change any of my code.

I have some logging on both the client and server to show which step in the handshake they are in.

Log output Java 8 - client

SSL Handshake Started
WRAP:OK -   BytesProduced=172  BytesConsumed=0
UNWRAP:OK - BytesProduced=0    BytesConsumed=2295
TASK
WRAP:OK -   BytesProduced=1815 BytesConsumed=0
WRAP:OK -   BytesProduced=269  BytesConsumed=0
WRAP:OK -   BytesProduced=6    BytesConsumed=0
WRAP:OK -   BytesProduced=85   BytesConsumed=0
UNWRAP:OK - BytesProduced=0    BytesConsumed=6
UNWRAP:OK - BytesProduced=0    BytesConsumed=85
SSL Handshake complete

Log output Java 8 - server

SSL Handshake Started
UNWRAP:OK - BytesProduced=0    BytesConsumed=172
TASK
WRAP:OK -   BytesProduced=2295 BytesConsumed=0
UNWRAP:OK - BytesProduced=0    BytesConsumed=1815
TASK
UNWRAP:OK - BytesProduced=0    BytesConsumed=269
TASK
UNWRAP:OK - BytesProduced=0    BytesConsumed=6
UNWRAP:OK - BytesProduced=0    BytesConsumed=85
WRAP:OK -   BytesProduced=6    BytesConsumed=6
WRAP:OK -   BytesProduced=85   BytesConsumed=0
SSL Handshake complete

Log output Java 11 - client

SSL Handshake Started
WRAP:OK -   BytesProduced=422  BytesConsumed=0
UNWRAP:OK - BytesProduced=0    BytesConsumed=160
TASK
WRAP:OK -   BytesProduced=6    BytesConsumed=0
UNWRAP:OK - BytesProduced=0    BytesConsumed=6
UNWRAP:OK - BytesProduced=0    BytesConsumed=2204
TASK
WRAP:OK -   BytesProduced=2067 BytesConsumed=0
SSL Handshake complete
UNWRAP:OK - BytesProduced=0    BytesConsumed=72

Log output Java 11 - server

SSL Handshake Started
UNWRAP:OK - BytesProduced=0    BytesConsumed=422
TASK
WRAP:OK -   BytesProduced=160  BytesConsumed=0
WRAP:OK -   BytesProduced=6    BytesConsumed=0
WRAP:OK -   BytesProduced=2204 BytesConsumed=0
UNWRAP:OK - BytesProduced=0    BytesConsumed=6
UNWRAP:OK - BytesProduced=0    BytesConsumed=2067
TASK
WRAP:OK -   BytesProduced=72    BytesConsumed=0
SSL Handshake complete

Code for handshaking

engine.beginHandshake();
HandshakeStatus hs = engine.getHandshakeStatus();
while(hs != HandshakeStatus.FINISHED && hs != HandshakeStatus.NOT_HANDSHAKING){
  switch(hs){
    case NEED_WRAP:
      SSLEngineResult res = engine.wrap(myAppData, myNetData)
      hs = res.getHandshakeStatus();
      switch(res.getStatus()){
        case OK:
          // write myNetData
        case BUFFER_OVERFLOW:
          // increase size of myNetData
        case BUFFER_UNDERFLOW:
          // throw exception
        case CLOSED:
          // clean up
        default:
          // throw illegal state exception
      }
      break;
    case NEED_UNWRAP:
      boolean complete = false;
      while(!complete){
        /*
         * First handle any encrypted data left on buffer
         * If there is none, read in more
         */
        if(peerNetData.position() > 0 || channel.read(peerNetData) > 0){
          peerNetData.flip();
          res = engine.unwrap(peerNetData, peerAppData);
          hs = res.getHandshakeStatus();
          switch(res.getStatus()){
            case OK:
              complete = true;
              peerNetData.compact();
              break;
            case BUFFER_UNDERFLOW:
              // if buffer is full, increase size
              // if buffer isn't full, compact and read
            case BUFFER_OVERFLOW:
              // increase size of peerAppData
            case CLOSED:
              // cleanup
            default:
              // throw illegal state exception
          }
        }
      }
      break;
    case NEED_TASK:
      // Run task
      hs = engine.getHandshakeStatus();
      break;
    case FINISHED:
      break;
    case NOT_HANDSHAKING:
      break;
    default:
      // illegal state
  }
}

Unfortunately, my code resides in an air-gapped environment, so pasting it here is not easy. I typed it in by hand so parenthesis and tabs might not line up.

The main point is the hs = res.getHandshakeStatus(...) returns FINISHED on the client machine after the wrap of 2067 bytes when it seems like it should return NEED_UNWRAP. If I change it to hs = engine.getHandshakeStatus(), it returns NOT_HANDSHAKING.

On the server machine, hs = engine.getHandshakeStatus() returns NEED_WRAP after running that last task causing it to WRAP that last 72 bytes.

Why does the SSLEngine on my client machine give me a handshake status of "FINISHED" when there are still 72 bytes of data to UNWRAP from the server? Has anyone else had any issues with custom handshake logic for Java 11?

Answer 1

The JavaDoc of SSLEngine has a note about concurrency at the end of the class description.

Thus I assume your issue looks like a concurrency issue based on a race condition. It can differ how wrap and unwrap functions are called simultaneously after you migrated from JDK 8 to JDK 11. The example code you supplied doesn't have any synchronized statements.

If you synchronize both function calls with a shared object the handshake should finish every time after the last unwrap.