Transitive Dependency: Using Elasticsearch Rest High Client problem in AEM

Question

I am trying to use Java High Level Rest Client in Adobe Experience Manager to finish project of comparison between Lucene, Solr and Elasticsearch search engines.

I am having some problems with elasticsearh implementation. Here is the code:

• Dependency in the parent pom.xml (the same is defined in core pom.xml)

  <!-- Elasticseach dependencies -->
  <dependency>
      <groupId>org.elasticsearch.client</groupId>
      <artifactId>elasticsearch-rest-high-level-client</artifactId>
      <version>7.4.0</version>
  </dependency>
  

• The only line of code that I am using that is from dependencies above

  try (RestHighLevelClient client = new 
  RestHighLevelClient(RestClient.builder(new HttpHost(server, port, 
  protocol),
    new HttpHost(server, secondPort, protocol)));)
  {
  
  }
  catch (ElasticsearchException e)
  {
      LOG.error("Exception: " + e);
  }
  

protocol = "http", server = "localhost", port = 9200, secondPort = 9201

• Error

• Dependencies from IntelliJ

I know that there is usually problem with dependencies versions, but all are 7.4.0 in this case. Also elasticsearch 7.4.0v is running locally on 3 nodes.

This project is done on We.Retail project so it is easy to replicate. Also all the code with this error is available here: https://github.com/tadijam64/search-engines-comparison-on-we-retail/tree/elasticsearch-integration AEM 6.4v.

Any info or idea is appreciated.

UPDATE I tried with adding the following to embed these dependencies externally since they are not OSGi dependencies:

<build>
        <plugins>
            <plugin>
                <groupId>org.apache.felix</groupId>
                <artifactId>maven-scr-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.apache.felix</groupId>
                <artifactId>maven-bundle-plugin</artifactId>
                <extensions>true</extensions>
                <configuration>
                    <instructions>
                        <Embed-Dependency>org.apache.servicemix.bundles.solr-solrj, log4j, noggit, zookeeper,
                            elasticsearch-rest-high-level-client
                        </Embed-Dependency>
                        <Embed-Transitive>true</Embed-Transitive>
                        <Embed-Directory>OSGI-INF/lib</Embed-Directory>
                        <Export-Package>we.retail.core.model*</Export-Package>
                        <Import-Package>
                            *;resolution:=optional
                        </Import-Package>
                        <Private-Package>we.retail.core*</Private-Package>
                        <Sling-Model-Packages>
                            we.retail.core.model
                        </Sling-Model-Packages>
                    </instructions>
                </configuration>
            </plugin>
        </plugins>
    </build>

The error remains. I also tried adding it to the "export-package", but nothing helps.

And by Elasticsearch documentation, all I need to use Elasticsearch is

<dependency>
    <groupId>org.elasticsearch.client</groupId>
    <artifactId>elasticsearch-rest-high-level-client</artifactId>
    <version>7.4.0</version>
</dependency>

but then NoClassDefFoundErrors occurs. It seems like a problem with transitive dependencies maybe. Not sure, but any idea is appreciated.

Some other suggestions can be found here: https://forums.adobe.com/thread/2653586

I have also tried adding it's transitive dependencies like org.elasticsearch and org.elasticsearch.client, but it does not work. The same error, just other class.

AEM version 6.4, Java version: jdk1.8.0_191.jdk

Answer 1

So my guess was right, transitive dependencies were not included altho <Embed-Transitive>true</Embed-Transitive> exists.

The following is necessary when running elasticsearch as a search engine on AEM the problem:

• I have added all transitive dependencies in pom.xml (versions are defined in parent/pom.xml):

    <!-- Elasticsearch -->
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch-x-content</artifactId>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.plugin</groupId>
            <artifactId>rank-eval-client</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-imaging</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.plugin</groupId>
            <artifactId>lang-mustache-client</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpasyncclient</artifactId>
        </dependency>
  

It is important to add all third-party dependencies as <Embed-Dependency> inside maven-bundle-plugin like this:

    <plugin>
        <groupId>org.apache.felix</groupId>
        <artifactId>maven-bundle-plugin</artifactId>
        <extensions>true</extensions>
        <configuration>
            <instructions>
                <Embed-Dependency>org.apache.servicemix.bundles.solr-solrj, noggit,
                    elasticsearch-rest-high-level-client,
                    elasticsearch,
                    elasticsearch-rest-client,
                    elasticsearch-x-content,
                    elasticsearch-core,
                    rank-eval-client,
                    lang-mustache-client,
                    httpasyncclient;
                </Embed-Dependency>
                <Embed-Transitive>true</Embed-Transitive>
                <Embed-Directory>OSGI-INF/lib</Embed-Directory>
                <Export-Package>we.retail.core.model*</Export-Package>
                <Import-Package>
                    *;resolution:=optional
                </Import-Package>
                <Private-Package>
                    we.retail.core*
                </Private-Package>
                <Sling-Model-Packages>
                    we.retail.core.model
                </Sling-Model-Packages>
                <_fixupmessages>"Classes found in the wrong directory";is:=warning</_fixupmessages>
            </instructions>
        </configuration>
    </plugin>

Important to notice:

• All third-party dependencies (the ones outside of OSGi) must be included in the "Embed-Dependency"
• "Embed-Transitive" must be set to true to include transitive dependencies
• "Import-Package" must include "*;resolution:=optional" to exclude all dependencies that could not be resolved so that the program can run normally
• For some reason, there was an error in compile time when "elasticsearch" dependency was added which is not important for this task, so I've decided to ignore it this way:

<_fixupmessages>"Classes found in the wrong directory";is:=warning</_fixupmessages>

Though challenging, I finally resolved it. There are many similar or the same problems on Google, so I hope this will help someone. Thanks to everyone that tried to help.