Elasticsearch Rest Client Still Giving IOException : Too Many Open Files

Question

This is a follow up to the solution which was provided to me on this previous post:

How to Properly Close Raw RestClient When Using Elastic Search 5.5.0 for Optimal Performance?

This same exact error message came back!

2017-09-29 18:50:22.497 ERROR 11099 --- [8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint   : Socket accept failed

java.io.IOException: Too many open files
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[na:1.8.0_141]
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) ~[na:1.8.0_141]
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) ~[na:1.8.0_141]
    at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:453) ~[tomcat-embed-core-8.5.15.jar!/:8.5.15]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]

2017-09-29 18:50:23.885  INFO 11099 --- [Thread-3] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@5387f9e0: startup date [Wed Sep 27 03:14:35 UTC 2017]; root of context hierarchy
2017-09-29 18:50:23.890  INFO 11099 --- [Thread-3] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase 2147483647
2017-09-29 18:50:23.891  WARN 11099 --- [Thread-3] o.s.c.support.DefaultLifecycleProcessor  : Failed to stop bean 'documentationPluginsBootstrapper'

    ... 7 common frames omitted

2017-09-29 18:50:53.891  WARN 11099 --- [Thread-3] o.s.c.support.DefaultLifecycleProcessor  : Failed to shut down 1 bean with phase value 2147483647 within timeout of 30000: [documentationPluginsBootstrapper]
2017-09-29 18:50:53.891  INFO 11099 --- [Thread-3] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown
2017-09-29 18:50:53.894  INFO 11099 --- [Thread-3] com.app.controller.SearchController  : Closing the ES REST client

I tried using the solution from the previous post.

ElasticsearchConfig:

@Configuration
public class ElasticsearchConfig {

@Value("${elasticsearch.host}")
private String host;

@Value("${elasticsearch.port}")
private int port;

@Bean
public RestClient restClient() {
    return RestClient.builder(new HttpHost(host, port))
    .setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
        @Override
        public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {
            return requestConfigBuilder.setConnectTimeout(5000).setSocketTimeout(60000);
        }
    }).setMaxRetryTimeoutMillis(60000).build();
}

SearchController:

@RestController
@RequestMapping("/api/v1")
public class SearchController {

    @Autowired
    private RestClient restClient;

    @RequestMapping(value = "/search", method = RequestMethod.GET, produces="application/json" )
    public ResponseEntity<Object> getSearchQueryResults(@RequestParam(value = "criteria") String criteria) throws IOException {

        // Setup HTTP Headers
        HttpHeaders headers = new HttpHeaders();
        headers.add("Content-Type", "application/json");

        // Setup query and send and return ResponseEntity...

        Response response = this.restClient.performRequest(...);
    }

    @PreDestroy
    public void cleanup() {
        try {
            logger.info("Closing the ES REST client");
            this.restClient.close();
        } 
        catch (IOException ioe) {
            logger.error("Problem occurred when closing the ES REST client", ioe);
        }
    }
}    

pom.xml:

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.4.RELEASE</version>
</parent>

<dependencies>
    <!-- Spring -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>

    <!-- Elasticsearch -->
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch</artifactId>
        <version>5.5.0</version>
    </dependency>

    <dependency>
        <groupId>org.elasticsearch.client</groupId>
        <artifactId>transport</artifactId>
        <version>5.5.0</version>
    </dependency>

    <!-- Apache Commons -->
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.6</version>
    </dependency>

    <!-- Log4j -->
    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.17</version>
    </dependency>
</dependencies>

This makes me think that the RestClient was never explicitly closing the connection, in the first place...

And this is surprising since my Elasticsearch Spring Boot based Microservice is load balanced on two different AWS EC-2 Servers.

That exception occurreded like 2000 times reported by the log file and only in the end did the preDestroy() close the client. See the INFO from the @PreDestroy() cleanup method being logged at the end of the StackTrace.

Do I need to explicitly put a finally clause inside the SearchController and close the RestClient connection explicitly?

It's really critical that this IOException doesn't happen again because this Search Microservice is dependent on a lot of different mobile clients (iOS & Android).

Need this to be fault tolerant and scalable... Or, at the very least, not to break.

The only reason this is in the bottom of the log file:

2017-09-29 18:50:53.894  INFO 11099 --- [Thread-3] com.app.controller.SearchController : Closing the ES REST client

Is because I did this:

kill -3 jvm_pid

Should I keep the @PreDestory cleanup() method but change the contents of my SearchController.getSearchResults() method to reflect something like this:

@RequestMapping(value = "/search", method = RequestMethod.GET, produces="application/json" )
public ResponseEntity<Object> getSearchQueryResults(@RequestParam(value = "criteria") String criteria) throws IOException {
    // Setup HTTP Headers
    HttpHeaders headers = new HttpHeaders();
    headers.add("Content-Type", "application/json");

    // Setup query and send and return ResponseEntity...
    Response response = null;

    try {
        // Submit Query and Obtain Response
        response = this.restClient.performRequest("POST", endPoint, Collections.singletonMap("pretty", "true"), entity);
    } 
    catch(IOException ioe) {
        logger.error("Exception when performing POST request " + ioe);
    }
    finally {
        this.restClient.close();
    }
    // return response as EsResponse();
}

This way the RestClient connection is always closing...

Would appreciate if someone can help me with this.

Answer 1

From my point of view, there are few thing that you are doing wrong but I will go directly to the solution.

I'm not going to write the full solution (in fact, I didn't execute or test anything), but the important is to understand it. Also, it is better if you move all related with data access to another layer. Anyway, this is only an example so the design is not perfect.

Step 1: Import the right library.

Practically the same as your example. I updated the example to use the last client library recommended in version 5.6.2

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.acervera</groupId>
  <artifactId>elastic-example</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>elastic-example</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <es.version>5.6.2</es.version>
  </properties>

  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.4.RELEASE</version>
  </parent>

  <dependencies>
    <!-- Spring -->
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
    </dependency>

    <!-- Elasticsearch -->
    <dependency>
      <groupId>org.elasticsearch</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${es.version}</version>
    </dependency>

<dependency>
  <groupId>org.elasticsearch.client</groupId>
  <artifactId>elasticsearch-rest-high-level-client</artifactId>
  <version>${es.version}</version>
</dependency>

    <!-- Log4j -->
    <dependency>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
      <version>1.2.17</version>
    </dependency>
  </dependencies>
</project>

Step 2: Create and shutdown the client in the bean factory.

In the bean factory, create and destroy it. You can reuse the same client.

import org.apache.http.HttpHost;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import java.io.IOException;

@Configuration
public class ElasticsearchConfig {


    // Here all init stuff with @Value(....)


    RestClient lowLevelRestClient;
    RestHighLevelClient client;

    @PostConstruct
    public void init() {
        lowLevelRestClient = RestClient.builder(new HttpHost("host", 9200, "http")).build();
        client = new RestHighLevelClient(lowLevelRestClient);
    }

    @PreDestroy
    public void destroy() throws IOException {
        lowLevelRestClient.close();
    }

    @Bean
    public RestHighLevelClient getClient() {
        return client;
    }

}

Step 3: Execute the query using the Java Transport Client.

Use the Java Transport Client to execute the query.

import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import java.io.IOException;

@RestController
@RequestMapping("/api/v1")
public class SearchController {

    @Autowired
    private RestHighLevelClient client;

    @RequestMapping(value = "/search", method = RequestMethod.GET, produces="application/json" )
    public ResponseEntity<Tweet> getSearchQueryResults(@RequestParam(value = "criteria") String criteria) throws IOException {

        // This is only one example. Of course, this logic make non sense and you are going to put it in a DAO
        // layer with more logical stuff
        SearchRequest searchRequest = new SearchRequest();
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.query(QueryBuilders.matchAllQuery());
        SearchResponse searchResponse = client.search(searchRequest);

        if(searchResponse.getHits().totalHits > 0) {
            SearchHit searchHit = searchResponse.getHits().iterator().next();

            // Deserialize to Java. The best option is to use response.getSource() and Jackson
            // This is other option.
            Tweet tweet = new Tweet();
            tweet.setId(searchHit.getField("id").getValue().toString());
            tweet.setTittle(searchHit.getField("tittle").getValue().toString());
            return ResponseEntity.ok(tweet);
        } else {
            return ResponseEntity.notFound().build();
        }

    }

}

Also, use a bean to build the response.

public class Tweet {

    private String id;

    private String tittle;

    public String getTittle() {
        return tittle;
    }

    public void setTittle(String tittle) {
        this.tittle = tittle;
    }

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    // Here rest of bean stuff (equal, hash, etc) or Lombok
}

Step: 4

Enjoy Elasticsearch!

Notes: Java REST Client [5.6] » Java High Level REST Client

PS. It is necessary to refactor the example. It is only to understand the way.