Menu
I'm playing around with Storm. This is a topology I'm using:
builder.setSpout("word", new RandomSentenceSpout(), 3);
builder.setBolt("exclaim1", new ExclamationBolt(), 6).shuffleGrouping("word");
I thought Storm would spawn 9 executors (3 spouts + 6 bolts) for this topology, but when I actually run it, I can see 11 executors are running.
What are those 2 extra executors?
680
Storm offers this and additional finer-grained ways to increase the parallelism of a Storm topology: Increase the number of workers. Increase the number of executors. Increase the number of tasks.
A worker process belongs to a specific topology and may run one or more executors for one or more components (spouts or bolts) of this topology. A running topology consists of many such processes running on many machines within a Storm cluster. An executor is a thread that is spawned by a worker process.
A Storm topology is analogous to a MapReduce job. One key difference is that a MapReduce job eventually finishes, whereas a topology runs forever (or until you kill it, of course). A topology is a graph of spouts and bolts that are connected with stream groupings.
edited Mar 25, 2020 by Praveen_1998. Apache storm cluster is similar to the Hadoop cluster but Storm use topologies instead of MapReduce jobs. In the storm, we need to terminate the topology otherwise it will run forever unlike MapReduce jobs. Apache storm cluster will have one master node and many worker nodes.
They are acker bolts which are responsible for managing acknowledgment mechanism. There is 2 ackers in your topology and each bolt task is equal to one executor by default.
storm uses acker as a bolt task (executor) and if we don't set Number of ackers, it will run some of them in the topology. if you want to exactly manage the number of executors, use the following :
Config conf = new Config();
conf.setNumAckers(1);
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
