Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

ClassNotFound error when running storm-starter topologies in local mode (Win10, OS X)

I'm trying to debug Storm Topology (on Storm v 1.0.0) under windows via:

TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new RandomIntegerSpout());
builder.setBolt("partialsum", new StatefulSumBolt("partial"), 1).shuffleGrouping("spout");
builder.setBolt("printer", new PrinterBolt(), 2).shuffleGrouping("partialsum");
builder.setBolt("total", new StatefulSumBolt("total"), 1).shuffleGrouping("printer");

Config conf = new Config();
conf.setDebug(false);
LocalCluster cluster = new LocalCluster();
StormTopology topology = builder.createTopology();
cluster.submitTopology("test", conf, topology);

And get the following error (WordCount/Exclamation/Stateful or other topologies from storm-starter - does not matter):

java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.storm.daemon.acker
at org.apache.storm.utils.Utils.javaDeserialize(Utils.java:181) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.utils.Utils.getSetComponentObject(Utils.java:430) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.daemon.task$get_task_object.invoke(task.clj:74) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.daemon.task$mk_task_data$fn__66.invoke(task.clj:177) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.util$assoc_apply_self.invoke(util.clj:930) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.daemon.task$mk_task_data.invoke(task.clj:170) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.daemon.task$mk_task.invoke(task.clj:181) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.daemon.executor$mk_executor$fn__6149.invoke(executor.clj:371) ~[storm-core-1.0.0.jar:1.0.0]
at clojure.core$map$fn__4553.invoke(core.clj:2622) ~[clojure-1.7.0.jar:?]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$fn__6506.invoke(protocols.clj:101) ~[clojure-1.7.0.jar:?]
at clojure.core.protocols$fn__6452$G__6447__6465.invoke(protocols.clj:13) ~[clojure-1.7.0.jar:?]
at clojure.core$reduce.invoke(core.clj:6519) ~[clojure-1.7.0.jar:?]
at clojure.core$into.invoke(core.clj:6600) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.executor$mk_executor.invoke(executor.clj:372) ~[storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.daemon.worker$fn__6779$exec_fn__3235__auto__$reify__6781$iter__6786__6790$fn__6791.invoke(worker.clj:634) ~[storm-core-1.0.0.jar:1.0.0]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.7.0.jar:?]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.7.0.jar:?]
at clojure.lang.RT.seq(RT.java:507) ~[clojure-1.7.0.jar:?]
at clojure.core$seq__4128.invoke(core.clj:137) ~[clojure-1.7.0.jar:?]
at clojure.core$dorun.invoke(core.clj:3009) ~[clojure-1.7.0.jar:?]
at clojure.core$doall.invoke(core.clj:3025) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__6779$exec_fn__3235__auto__$reify__6781.run(worker.clj:634) ~[storm-core-1.0.0.jar:1.0.0]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_65]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_65]
at org.apache.storm.daemon.worker$fn__6779$exec_fn__3235__auto____6780.invoke(worker.clj:606) ~[storm-core-1.0.0.jar:1.0.0]
at clojure.lang.AFn.applyToHelper(AFn.java:178) ~[clojure-1.7.0.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__6779$mk_worker__6874.doInvoke(worker.clj:580) [storm-core-1.0.0.jar:1.0.0]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.supervisor$fn__7647.invoke(supervisor.clj:1200) [storm-core-1.0.0.jar:1.0.0]
at clojure.lang.MultiFn.invoke(MultiFn.java:251) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.supervisor$get_valid_new_worker_ids$iter__7208__7212$fn__7213.invoke(supervisor.clj:380) [storm-core-1.0.0.jar:1.0.0]
at clojure.lang.LazySeq.sval(LazySeq.java:40) [clojure-1.7.0.jar:?]
at clojure.lang.LazySeq.seq(LazySeq.java:49) [clojure-1.7.0.jar:?]
at clojure.lang.RT.seq(RT.java:507) [clojure-1.7.0.jar:?]
at clojure.core$seq__4128.invoke(core.clj:137) [clojure-1.7.0.jar:?]
at clojure.core$dorun.invoke(core.clj:3009) [clojure-1.7.0.jar:?]
at clojure.core$doall.invoke(core.clj:3025) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.supervisor$get_valid_new_worker_ids.invoke(supervisor.clj:367) [storm-core-1.0.0.jar:1.0.0]
at org.apache.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:428) [storm-core-1.0.0.jar:1.0.0]
at clojure.core$partial$fn__4527.invoke(core.clj:2492) [clojure-1.7.0.jar:?]
at org.apache.storm.event$event_manager$fn__909.invoke(event.clj:40) [storm-core-1.0.0.jar:1.0.0]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]

This behaves like if there is no jar-file in workers's working directory. But according to comment ";;in local mode there is no jar" in Nimbus.clj it does not seems to be wrong. In version 0.10.0 this problem does not occur. Any ideas?

The problem occurs when I try to run a debug in Intellij using maven-exec plugin with the following command line in configuration (like recommended here):

compile exec:java -Dstorm.topology=org.apache.storm.starter.WordCountTopology 

from directory, where topology's POM-file is situated.

UPD: the problem is definetely caused by unavailability of any (topology or storm-core) classes for worker thread while it's initialization. (Trying both WordCount and Exclamation topologies illustrates this). Playings with scope of "storm-core" dependency and intellij profile in topology's POM-s gave nothing.

like image 751
onetwothree12345 Avatar asked Apr 15 '16 14:04

onetwothree12345


2 Answers

I've been running my LocalCluster with sbt run, if I run it using java -jar fatjar.jar everything runs fine. For now I'm going to run with Intellij sorting out the classpath, no idea why sbt is behaving strangely when resolving the classpath. Anyone with any info please comment!

like image 91
Matthew Pickering Avatar answered Nov 05 '22 01:11

Matthew Pickering


I found some solution: use of standard Idea's building and running functionality works well for clear java topologies. But only until execution will run some shell-scripts (like splitsentece.py). Then it fails with file not found exception, so compilation without maven seems to be not fully functional

like image 25
onetwothree12345 Avatar answered Nov 05 '22 00:11

onetwothree12345