java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;

Question

I've been trying to connect kafka to elasticsearch using kafka-connect api.Kafka version is 0.11.0.0.These are the steps I followed:

1.Buiding Elasticsearch Connector:

https://github.com/confluentinc/kafka-connect-elasticsearch.git

2.Build the connector

$ cd kafka-connect-elasticsearch
$ mvn clean package

3.Finally running the script:

$ bin/connect-standalone.sh config/connect-standalone.properties config/elasticsearch-connect.properties

It throws the following exception:

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
    at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
    at org.reflections.Reflections.<init>(Reflections.java:126)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:221)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:198)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:159)
    at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:68)

Can't understand what's going wrong.

Answer 1

From experience, this error means that you are bringing an older version of guava earlier in your classpath. Connect worker requires guava >= 20 for org.reflections to work correctly.

kafka-connect-elasticsearch or any other connector that brings with it guava 18.0 or older will prohibit the worker from starting up. This error message means that the older guava jar was encountered first in the classpath.

Two solutions:

1. Indeed, as Hans Jespersen mentions, using classloading isolation by setting your plugin.path in Connect worker's configuration, will allow the connector to work as-is without interfering with the Connect framework.
2. If adding the connector to the CLASSPATH is your only option, make sure it's added after Kafka Connect's dependencies, so that the most recent guava will be picked up.

Answer 2

This appears to describe the answer for your problem https://github.com/confluentinc/kafka-connect-elasticsearch/issues/104

It's a little confusing, but after you build the connector there are a number of things in the target directory. The kafka-connect-elasticsearch-.jar is only the JAR file with the connector code, but that doesn't include all the libraries. One of those directories in the target directory, namely target/kafka-connect-elasticsearch-*-development/share/java/kafka-connect-elasticsearch/, does contain all the libraries. Add this directory to the Kafka Connect worker's classpath, or copy all of those JAR files into a directory that is already on the classpath.