Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Error when creating a StreamingContext

I open the spark shell

spark-shell --packages org.apache.spark:spark-streaming-kafka_2.10:1.6.0

Then I want to create a streaming context

import org.apache.spark._
import org.apache.spark.streaming._


val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount").set("spark.driver.allowMultipleContexts", "true")
val ssc = new StreamingContext(conf, Seconds(1))

I run into a exception:

org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at:
like image 557
Stefan Papp Avatar asked Feb 23 '16 21:02

Stefan Papp


People also ask

What happens when we apply a flatMap transformation to a DStream?

flatMap is a DStream operation that creates a new DStream by generating multiple new records from each record in the source DStream. In this case, each line will be split into multiple words and the stream of words is represented as the words DStream.

What is block interval in Spark Streaming?

spark.streaming.blockInterval : Interval at which data received by Spark Streaming receivers is chunked into blocks of data before storing them in Spark. This is when using receiver bases approach - Receiver-based Approach.

What is Spark Streaming used for?

Spark Streaming is an extension of the core Spark API that allows data engineers and data scientists to process real-time data from various sources including (but not limited to) Kafka, Flume, and Amazon Kinesis. This processed data can be pushed out to file systems, databases, and live dashboards.

How does Spark approach stream processing?

Spark streaming takes live data streams as input and provides as output batches by dividing them. These streams are then processed by the Spark engine and the final stream results in batches.


1 Answers

When you open the spark-shell, there is already a streaming context created. It is called sc, meaning you do not need to create a configure object. Simply use the existing sc object.

val ssc = new StreamingContext(sc,Seconds(1))

instead of var we will use val

like image 200
Stefan Papp Avatar answered Sep 28 '22 03:09

Stefan Papp