Exception: 'writeStream' can be called only on streaming Dataset/DataFrame

Question

Trying to create a test for spark data streaming writeStream function as shown below:

SparkSession spark = SparkSession.builder().master("local").appName("spark 
session").getOrCreate()

val lakeDF = spark.createDF(List(("hi")), List(("word", StringType, true)))

lakeDF.writeStream
  .trigger(Trigger.Once)
  .format("parquet")
  .option("checkpointLocation", checkpointPath)
  .start(dataPath)

But I am getting following exception: org.apache.spark.sql.AnalysisException: 'writeStream' can be called only on streaming Dataset/DataFrame;

I am very new to spark streaming, please let me know how can i create a streaming dataframe/convert the above regular dataframe into streaming dataframe for my test suite.

Answer 1

In Spark Structured Streaming dataframes/datasets are created out stream using readStream on SparkSession. If the dataframe/dataset are not created using stream then you are not allowed store using writeStream.

So create the dataframes/datasets using readStream and store the dataframes/datasets using writeStream

 val kafkaStream = sparkSession.readStream.format("kafka")
.option("kafka.bootstrap.servers", "kafka-broker-hostname:port")
.option("subscribe", "topicname")
.load()