How to Properly Close Raw RestClient When Using Elastic Search 5.5.0 for Optimal Performance?

Question

Am using a Spring Boot 1.5.4.RELEASE Microservice to connect to an ElasticSearch 5.5.0 instance using the low level Rest Client that ElasticSearch provides.

pom.xml

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.4.RELEASE</version>
</parent>

<dependencies>
    <!-- Spring -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>

    <!-- Elasticsearch -->
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch</artifactId>
        <version>5.5.0</version>
    </dependency>

    <dependency>
        <groupId>org.elasticsearch.client</groupId>
        <artifactId>transport</artifactId>
        <version>5.5.0</version>
    </dependency>

    <!-- Apache Commons -->
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.6</version>
    </dependency>

    <!-- Jackson -->
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-core</artifactId>
        <version>2.8.9</version>
    </dependency>

    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.8.9</version>
    </dependency>

    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-annotations</artifactId>
        <version>2.8.9</version>
    </dependency>

    <!-- Log4j -->
    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.17</version>
    </dependency>

    <!-- JUnit -->
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.11</version>
        <scope>test</scope>
    </dependency>

    <!-- Swagger -->
    <dependency>
        <groupId>io.springfox</groupId>
        <artifactId>springfox-swagger2</artifactId>
        <version>2.6.1</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>io.springfox</groupId>
        <artifactId>springfox-swagger-ui</artifactId>
        <version>2.6.1</version>
        <scope>compile</scope>
    </dependency>
</dependencies>

Everything is setup correctly but after a bunch of hits, client apps were reporting an HTTP 500 error and this is what appeared in the log files:

java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method) ~[na:1.8.0_141]
        at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65) ~[na:1.8.0_141]
        at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) ~[na:1.8.0_141]
        at java.nio.channels.Selector.open(Selector.java:227) ~[na:1.8.0_141]
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.<init>(AbstractMultiworkerIOReactor.java:142) ~[httpcore-nio-4.4.5.jar!/:4.4.5]
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.<init>(DefaultConnectingIOReactor.java:79) ~[httpcore-nio-4.4.5.jar!/:4.4.5]
        at org.apache.http.impl.nio.client.IOReactorUtils.create(IOReactorUtils.java:43) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
        at org.apache.http.impl.nio.client.HttpAsyncClientBuilder.build(HttpAsyncClientBuilder.java:666) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
        at org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:202) ~[rest-5.5.0.jar!/:5.5.0]
        at org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:180) ~[rest-5.5.0.jar!/:5.5.0]
        at com.myapp.controller.SearchController.getSearchQueryResults(SearchController.java:94) ~[classes!/:1.0]

Inside SearchController (the second line after the // comment is line 94):

@RestController
@RequestMapping("/api/v1")
public class SearchController {

    @RequestMapping(value = "/search", method = RequestMethod.GET, produces="application/json" )
    public ResponseEntity<Object> getSearchQueryResults(@RequestParam(value = "criteria") String criteria) throws IOException {

        // Setup HTTP Headers
        HttpHeaders headers = new HttpHeaders();
        headers.add("Content-Type", "application/json");

        // Setup RestClient
        RestClient restClient = RestClient.builder(new HttpHost("localhost", 9200))
        .setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
            @Override
            public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {
                return requestConfigBuilder.setConnectTimeout(5000).setSocketTimeout(60000);
            }
        }).setMaxRetryTimeoutMillis(60000).build();

        // Setup query and send and return ResponseEntity...

    }
}

Its obvious that what it was never closed after calling the restClient.performRequest() method...

So, I put this into my code:

Response response = null;
try {
   // Submit Query and Obtain Response
   response = restClient.performRequest("POST", endPoint,  Collections.singletonMap("pretty", "true"), entity);
}
catch (IOException e) {
   LOG.error("\n\n\tException: " + e + "\n\n");
   e.printStackTrace();
}
finally {
   restClient.close();
}

Read on Elastic Search's documentation that the RestClient class is thread-safe...

Also, read about the restClient.performRequestAsync() method but am somewhat inexperienced with threads and the description inside the documentation is vague.

Question(s):

1. Is my solution the best way to handle and close a bunch of socket resources?

2. Would appreciate if someone could show me a better way to use the low level RestClient with Elastic Search in sense that it won't cause the same issue with the socket resources not being freed resulting in an HTTP 500. Should I be using restClient.performRequestAsync? Could someone please provide an example?

Thank you for taking the time to read this...

Answer 1

It's is not a good practice to create a RestClient on every single request. You should create a single instance via a configuration bean like the one below:

@Configuration
public class ElasticsearchConfig {

    @Value("${elasticsearch.host}")
    private String host;

    @Value("${elasticsearch.port}")
    private int port;

    @Bean
    public RestClient restClient() {
        return RestClient.builder(new HttpHost(host, port))
        .setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
            @Override
            public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {
                return requestConfigBuilder.setConnectTimeout(5000).setSocketTimeout(60000);
            }
        }).setMaxRetryTimeoutMillis(60000).build();
    }
}

And then in your SearchController class you can inject it like this (and also add a cleanup method to close the restClient instance when your container goes down):

@RestController
@RequestMapping("/api/v1")
public class SearchController {

    @Autowired
    private RestClient restClient;

    @RequestMapping(value = "/search", method = RequestMethod.GET, produces="application/json" )
    public ResponseEntity<Object> getSearchQueryResults(@RequestParam(value = "criteria") String criteria) throws IOException {

        // Setup HTTP Headers
        HttpHeaders headers = new HttpHeaders();
        headers.add("Content-Type", "application/json");

        // Setup query and send and return ResponseEntity...

        Response response = this.restClient.performRequest(...);

    }

    @PreDestroy
    public void cleanup() {
        try {
            logger.info("Closing the ES REST client");
            this.restClient.close();
        } catch (IOException ioe) {
            logger.error("Problem occurred when closing the ES REST client", ioe);
        }
    }

}