ES Rest High Level Client throws SocketTimeoutException after being idle for sometime

Question

RestHighLevelClient is used to connect to ES 6.4(hosted on AWS) in a spring-boot app. When the app goes idle for some time, and request arrives, then the RestHighLevelClient throws SocketTimeoutException:

[Request processing failed; nested exception is org.springframework.data.elasticsearch.ElasticsearchException: Error while bulk for request: org.elasticsearch.action.bulk.BulkRequest@21511b6c] w
java.net.SocketTimeoutException: 5,000 milliseconds timeout on connection http-outgoing-38 [ACTIVE]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.11.jar!/:4.4.11]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

RestHighLevelClient is created using:

    @Bean
    RestHighLevelClient client() {

        ClientConfiguration clientConfiguration = ClientConfiguration.builder()
                .connectedTo(elasticsearchHostAndPort)
                .build();

        return RestClients.create(clientConfiguration).rest();
    }

spring-data-elasticsearch version 3.2.0.M2 is used.

Any hints/workarounds?

Answer 1

I have also tried setting the connection/socket timeout to 0, as suggested here and in other places. It didn't help eventually.

There is another solution/workaround, suggested by spring-data-elasticsearch in https://jira.spring.io/browse/DATAES-789. It simply perform an internal retry in case of such exception. It does not really solve the issue, but your client will not get and error. Instead, the first request after idle time will take additional 5 seconds (or whatever timeout your configured).

If you use Springboot-data-elasticsearch version 4+ (staring with Springboot 2.3.0), then you can apply the solution.

I can confirm the following solution is working (with the limitations I mentioned above):

@Configuration
public class ElasticSearchRestClientConfig extends AbstractElasticsearchConfiguration {

    @Autowired
    private RestHighLevelClient restHighLevelClient;

    @Override
    public RestHighLevelClient elasticsearchClient() {
        return restHighLevelClient;
    }

    @Bean
    @Override
    public ElasticsearchCustomConversions elasticsearchCustomConversions() {
        return new ElasticsearchCustomConversions();
    }

    @Override
    public ElasticsearchOperations elasticsearchOperations(ElasticsearchConverter elasticsearchConverter) {
        return new ElasticsearchRestTemplate(elasticsearchClient(), elasticsearchConverter) {
            @Override
            public <T> T execute(ClientCallback<T> callback) {
                int retryCount = 0;
                T t = null;
                while (retryCount <= RestClientBuilder.DEFAULT_MAX_CONN_PER_ROUTE && t == null) {
                    try {
                        t = super.execute(callback);
                    } catch (DataAccessResourceFailureException e) {
                        // retry
                        if (e.getCause() != null && (e.getCause().getCause() instanceof SocketTimeoutException) &&
                                (retryCount < RestClientBuilder.DEFAULT_MAX_CONN_PER_ROUTE)) {
                            retryCount++;
                            log.warn("Elasticsearch client - performing retry {} after caught DataAccessResourceFailureException: {}", retryCount, e.getMessage());
                        }
                        else {
                            throw e;
                        }
                    }
                }
                return t;
            }
        };
    }
 

Answer 2

In RestClientBuilder.createHttpClient() the defaults for socket timeout and connection timeout are set to 30 and 10 seconds.

You can override this defaults by implementing RestClientBuilder.RequestConfigCallback and calling setRequestConfigCallback(...) on your RestHighLevelClient

We did something like

@Override
public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder builder) {
    return builder.setSocketTimeout(socketTimeout); // try to prevent SocketTimeoutException
}