Spark accumulator not displayed in spark WebUI

Question

I'm using spark streaming. According to the Spark Programming Guide (see http://spark.apache.org/docs/latest/programming-guide.html#accumulators), named accumulators will be displayed in the WebUI as below: Unfortunately, I cannot find this anywhere. I am registering the accumulators like this (Java):

LongAccumulator accumulator = new LongAccumulator();    
ssc.sparkContext.sc().register(accumulator, "my accumulator");

I am using Spark 2.0.0.

Answer 1

I do not have a working streaming example but in non streaming example this UI could be found at the stages tab when choosing a specific stage. Also, I generally create the accumulator like this:

val accum = sc.longAccumulator("My Accumulator")

The equivalent in for spark streaming would probably be to replace sc with ssc.SparkContext

Answer 2

It worked for me. Below is my sample code

Accumulator<Integer> spansWritten = jsc.sparkContext().intAccumulator(0,"Spans_Written");
JavaDStream<Span> dStream = SourceFactory.getSource().createStream(jsc)
    .map( s -> {
      spansWritten.add(1);
      return s;
    });

However, when I tried to use them inside a Decoder while creating stream for kafka, it didn't show up in the UI.

Here is how it looks in the UI (select stages tab from the top, and click on one of the stage) screen shot