I've installed Flume and Hadoop manually (I mean, not CDH) and I'm trying to run the twitter example from Cloudera.
In the apache-flume-1.5.0-SNAPSHOT-bin
directory, I start the agent with the following command:
bin/flume-ng agent -c conf -f conf/twitter.conf -Dflume.root.logger=DEBUG,console -n TwitterAgent
My conf/twitter.conf
file uses the logger as the sink. The conf/flume-env.sh
assigns to CLASSPATH the flume-sources-1.0-SNAPSHOT.jar
that contains the definition of the twitter source. The resulting output is:
(...) [ERROR org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)] Unable to start EventDrivenSourceRunner: { source:com.cloudera.flume.source.TwitterSource{name:Twitter,state:IDLE} } - Exception follows. java.lang.NoSuchMethodError:
twitter4j.FilterQuery.setIncludeEntities(Z)Ltwitter4j/FilterQuery;
at com.cloudera.flume.source.TwitterSource.start(TwitterSource.java:139)
The conflict results from a FilterQuery
class that is defined elsewhere in the flume lib and that does not contain the setIncludeEntities
method. For me, the file that contains this class is the twitter4j-stream-3.0.3.jar
and I cannot exclude the file from the classpath as suggested here.
I believe this experience was quite frustrating for you, for me it was for sure. The main problem is, both the files, flume-sources-1.0-SNAPSHOT.jar and twitter4j-stream-3.0.3.jar contains the same FilterQuery.class. That is why the conflict message is generated in the log file.
I am not a Java or Big Data expert, but I can give you an alternate to this problem. Download the Twitter4j-stream-2.6.6.jar or lower version from here and replacethe twitter4j-stream-3.0.3.jar. All the 3.X.X uses this class. After replacing, everything should work fine. But you may get some heap error after downloading huge amount of tweets. Please google the solution as it was resolved in 3.X.X files.
-Edit Also, please don't forget to download and replace all the twitter4j files in /usr/lib/flume-ng folder. Namely, twitter4j-media-support-2.2.6.jar, twitter4j-stream-2.2.6.jar and twitter4j-core-2.2.6.jar. Any mismatch related to version among these files will also create problem.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With