Jersey Client HTTPS performance issue

Question

I am using a JUnit test with JERSEY Client + HTTPS to test a secure web service running on Jetty. One of my calls to wr.get( ClientResponse.class) hangs for 10 seconds after every block of 9-10 requests. The equivalent Apache client code runs in a few milliseconds. If I switch Jetty to HTTP mode the problem disappears.

I am using Jersey bundle & client 1.14 and Jetty 6.1.26

The apache client works

@Test
public void testApacheClient() throws Exception
{
  HttpClient client = new HttpClient();
  ProtocolSocketFactory socketFactory = new EasySSLProtocolSocketFactory();
  Protocol https = new Protocol( "https", socketFactory, 443 );
  Protocol.registerProtocol( "https", https );

  //Finishes in < 1 second
  for ( int i = 0; i < 30; i++ )
  {
    GetMethod getMethod = new GetMethod( "https://localhost:8443/home" );
    client.executeMethod( getMethod );
    String entity = getMethod.getResponseBodyAsString();
    getMethod.releaseConnection();
  }
}

The Jersey client hangs

@Test
public void testJerseyClient() throws Exception

  HostnameVerifier hv = getHostnameVerifier();
  ClientConfig config = new DefaultClientConfig();
  SSLContext ctx = getSslContext();
  config.getProperties().put( HTTPSProperties.PROPERTY_HTTPS_PROPERTIES,
      new HTTPSProperties( hv, ctx ) );

  Client jerseyClient = Client.create( config );

  //Finishes in < 1 second
  for ( int i = 0; i < 30; i++ )
  {
    WebResource wr = jerseyClient.resource( "https://www.google.com" );
    ClientResponse cr = wr.get( ClientResponse.class );
    String entity = cr.getEntity( String.class );
    cr.close();
  }

  /* Pauses for 10 seconds after the 10th request, and subsequently after every 9th request.
     Debugging shows waiting at line 'ClientResponse cr = ...'
   */
  for ( int i = 0; i < 30; i++ )
  {
    WebResource wr = jerseyClient.resource( "https://localhost:8443/home" );
    ClientResponse cr = wr.get( ClientResponse.class );  //10 second pause after requests 9, 18, 27
    String entity = cr.getEntity( String.class ); //This is triggering the problem
    cr.close();
  }

  //Finishes in < 1 second
  for ( int i = 0; i < 30; i++ )
  {
    WebResource wr = jerseyClient.resource( "https://localhost:8443/home" );
    ClientResponse cr = wr.get( ClientResponse.class );
    cr.close();
  }
}

Retrieving the entity from ClientResponse seems to trigger the problem, but only in HTTPS mode and running against my web server (not google, facebook, etc). I have run Jersey ServletContainer and Struts ActionServlet on Jetty and the problem occurs with both. I've also run the web server on different machines on my subnet and unit tested from multiple machines.

Jersey HTTPS classes

private HostnameVerifier getHostnameVerifier() {
  HostnameVerifier hv = new HostnameVerifier() {
    public boolean verify( String arg0, SSLSession arg1 ) { return true; }
  };
  return hv;
}

private SSLContext getSslContext() throws Exception {      
  private final SSLContext sslContext = SSLContext.getInstance( "SSL" );
  sslContext.init( null, new TrustManager[] { new X509TrustManager() {
        public X509Certificate[] getAcceptedIssuers() { return new X509Certificate[0]; }
        public void checkClientTrusted( X509Certificate[] certs, String authType ) {}
        public void checkServerTrusted( X509Certificate[] certs, String authType ) {}
      }
    }, new SecureRandom()
  );
  return sslContext;
}

Jetty SSL connector. If I use a SelectChannelConnector and unit test with HTTP the problem goes away.

<Call name="addConnector">
  <Arg>
    <New class="org.mortbay.jetty.security.SslSocketConnector">
      <Set name="allowRenegotiate">true</Set>
      <Set name="Port">8443</Set>
      <Set name="reuseAddress">true</Set>
      <Set name="Host">mgmt-int</Set>
      <Set name="maxIdleTime">30000</Set>
      <Set name="handshakeTimeout">2000</Set>
      <Set name="keystore"><SystemProperty name="jetty.home" default="/usr/app/web-app"/>/conf/keystore.jetty</Set>
      <Set name="password">OBF:1vaaaaaaaaaaaaaaaaaaaa111111111v</Set>
      <Set name="keyPassword">OBF:1vaaaaaaaaaaaaaaaaaaaa111111111v</Set>
    </New>
  </Arg>
</Call>

Answer 1

What OS are these running on?

Just a guess, but it could be related to a slow /dev/random implementation (via SecureRandom).

Related discussion here:

How to solve performance problem with Java SecureRandom?

Since I can't always control JVM parameters in a web app, I've been using this as a workaround:

static {
    System.setProperty("java.security.egd", "file:///dev/urandom");
}

Don't know if that's strictly speaking recommended (but it solves the problem for me).