RestTemplate + ConnectionPoolTimeoutException: Timeout waiting for connection from pool

Question

I got this error all of a sudden in production while the application was not under any load.

The issue happened when my code tries to send the PUT message using spring rest template

Here is the code how I am initialing the restTemplate

private static final RestTemplate restTemplate = new RestTemplate(new HttpComponentsClientHttpRequestFactory());
{

    List<HttpMessageConverter<?>> messageConverters = new ArrayList<HttpMessageConverter<?>>();
    Jaxb2Marshaller marshaller = new Jaxb2Marshaller();
    marshaller.setClassesToBeBound(PaymentSession.class);
    MarshallingHttpMessageConverter marshallingHttpMessageConverter = new MarshallingHttpMessageConverter(marshaller, marshaller);
    marshallingHttpMessageConverter.setSupportedMediaTypes(Arrays.asList(MediaType.APPLICATION_XML, MediaType.TEXT_HTML));
    messageConverters.add(marshallingHttpMessageConverter);
    restTemplate.setMessageConverters(messageConverters);
}

Call to PUT

try {
    HttpHeaders headers = new HttpHeaders();
    headers.setContentType(MediaType.APPLICATION_XML);
    HttpEntity<PaymentSession> httpEntity = new HttpEntity<PaymentSession>(session, headers);

    restTemplate.exchange(baseUrl+"/v1/psps", HttpMethod.PUT, httpEntity, PaymentSession.class);

}catch(HttpClientErrorException e){
        logger.error("Exception..!!",e)
}

Exception stacktrace

Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:232)
at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:199)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:456)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:88)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:46)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:49)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:509)

Answer 1

I would suggest to configure HttpComponentsClientHttpRequestFactory instance being passed in the constructor of RestTemplate increasing defaultMaxPerRoute or maxPerRoute for the specific http route for which requests are timing out, increasing the pool size is not enough, as I mentioned in a comment, even if you set PoolingHttpClientConnectionManager.setMaxTotal() to 200, HttpComponentsClientHttpRequestFactory uses a defaultMaxPerRoute of 4, my guess would be in an attempt for a host route (scheme, host, port) not to hijack the connection pool)

...
public PoolingHttpClientConnectionManager poolingHttpClientConnectionManager() {
        PoolingHttpClientConnectionManager result = new PoolingHttpClientConnectionManager();
        result.setMaxTotal(this.httpHostConfiguration.getMaxTotal());
        // Default max per route is used in case it's not set for a specific route
        result.setDefaultMaxPerRoute(this.httpHostConfiguration.getDefaultMaxPerRoute());
        // and / or
        if (CollectionUtils.isNotEmpty(this.httpHostConfiguration.getMaxPerRoutes())) {
            for (HttpHostConfiguration httpHostConfig : this.httpHostConfiguration.getMaxPerRoutes()) {
                HttpHost host = new HttpHost(httpHostConfig.getHost(), httpHostConfig.getPort(), httpHostConfig.getScheme());
                // Max per route for a specific host route
                result.setMaxPerRoute(new HttpRoute(host), httpHostConfig.getMaxPerRoute());
            }
        }
        return result;
    }
  ...


@Configuration
@ConfigurationProperties(prefix = "httpConnPool")
public class HttpHostsConfiguration {

  private Integer maxTotal;
  private Integer defaultMaxPerRoute;
  private List<HttpHostConfiguration> maxPerRoutes;

  // Getters, Setters
...

application.yml

httpConnPool:
  maxTotal: 20
  defaultMaxPerRoute: 20
  maxPerRoutes:
    -
      scheme: http
      host: localhost
      port: 8800
      maxPerRoute: 20

I recently blog about Troubleshooting Spring's RestTemplate Requests Timeout where requests timing out were troubleshooted using JMeter and shell commands and fixed via configuration settings.