Why does starting StreamingContext fail with “IllegalArgumentException: requirement failed: No output operations registered, so nothing to execute”?

Question

I'm trying to execute a Spark Streaming example with Twitter as the source as follows:

public static void main (String.. args) {      SparkConf conf = new SparkConf().setAppName("Spark_Streaming_Twitter").setMaster("local");         JavaSparkContext sc = new JavaSparkContext(conf);                JavaStreamingContext jssc = new JavaStreamingContext(sc, new Duration(2));               JavaSQLContext sqlCtx = new JavaSQLContext(sc);                String[] filters = new String[] {"soccer"};          JavaReceiverInputDStream<Status> receiverStream = TwitterUtils.createStream(jssc,filters);             jssc.start();          jssc.awaitTermination();  } 

But I'm getting the following exception

Exception in thread "main" java.lang.AssertionError: assertion failed: No output streams registered, so nothing to execute     at scala.Predef$.assert(Predef.scala:179)     at org.apache.spark.streaming.DStreamGraph.validate(DStreamGraph.scala:158)     at org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:416)     at org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:437)     at org.apache.spark.streaming.api.java.JavaStreamingContext.start(JavaStreamingContext.scala:501)     at org.learning.spark.TwitterStreamSpark.main(TwitterStreamSpark.java:53) 

Any suggestion how to fix this issue?

Answer 1

When an output operator is called, it triggers the computation of a stream.

Without output operator on DStream no computation is invoked. basically you will need to invoke any of below method on stream

print() foreachRDD(func) saveAsObjectFiles(prefix, [suffix]) saveAsTextFiles(prefix, [suffix]) saveAsHadoopFiles(prefix, [suffix]) 

http://spark.apache.org/docs/latest/streaming-programming-guide.html#output-operations

you can also first apply any transformations and then output functions too if required.