Backpressure in storm

Question

In our storm 1.0.2 application we are facing out of memory exceptions.Upon debugging we saw that the Kafka spout was emitting too many messages to the bolts . The bolts were running at a capacity of almost 4.0. So is there a way to enable backpressure in storm so that the spout emits depending on the capacity in bolts. Tried enabling the topology.backpressure.enable to true but ran to this issue https://issues.apache.org/jira/browse/STORM-1949. We are using the out of the box implementation of KafkaSpout and extending the BaseRichBolt for our bolts .Our DAG is linear.

Answer 1

You can handle the back pressure of the KafkaSpout by setting the maxSpoutPending value in the topology configuration,

Config config = new Config();
config.setMaxSpoutPending(200); 
config.setMessageTimeoutSecs(100);

StormSubmitter.submitTopology("testtopology", config, builder.createTopology());

maxSpoutPending is the number of tuples that can be pending acknowledgement in your topology at a given time. Setting this property, will intimate the KafkaSpout not to consume any more data from Kafka unless the unacknowledged tuple count is less than maxSpoutPending value.