Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

Spark Context is not automatically created in Scala Spark Shell

Tags:

apache-spark

I read in a Spark book :

Driver programs access Spark through a SparkContext object, which represents a connection to a computing cluster. In the shell, a SparkContext is automatically created for you as the variable called sc. Try printing out sc to see its type

sc

When I enter sc, it gives me an error 20 value sc not found. Any idea why is sc not automatically created in my scala spark shell?

I try to manually create a sc and it gave me an error saying there is already a spark context in the JVM. Please see pic :

http://s30.photobucket.com/user/kctestingeas1/media/No%20Spark%20Context.jpg.html

I believe i am already in scala spark shell as you can see on the top of my cmd window indicating bin\spark-shell

Please advise. Thanks

like image 463
Nipponho Avatar asked Jan 31 '17 18:01

Nipponho

People also ask

Do we need to create Spark context in Spark shell?

The first thing a Spark program must do is to create a SparkContext object, which tells Spark how to access a cluster. To create a SparkContext you first need to build a SparkConf object that contains information about your application. Only one SparkContext may be active per JVM.

Where is Spark context created?

SparkContext is the entry point to any spark functionality. When we run any Spark application, a driver program starts, which has the main function and your SparkContext gets initiated here. The driver program then runs the operations inside the executors on worker nodes.

Which of the following is true of the Spark interactive shell?

What is true of the Spark Interactive Shell? It initializes SparkContext and makes it available. Provides instant feedback as code is entered, and allows you to write programs interactively.

How do I create a Spark session in Spark shell?

To create SparkSession in Scala or Python, you need to use the builder pattern method builder() and calling getOrCreate() method. If SparkSession already exists it returns otherwise creates a new SparkSession. SparkSession. builder() – Return SparkSession.

1 Answers

Hopefully you found the answer to your question, because I am encountering the same issue as well.

In the meantime, use this workaround. In the scala spark shell, enter:

  1. import org.apache.spark.SparkContext
  2. val sc = SparkContext.getOrCreate()

You then have access to sc.

like image 142
Tri Han Avatar answered Oct 25 '22 05:10

Tri Han
Sign in to Comment

Related questions

Enriching SparkContext without incurring in serialization issues
spark reading large file
Using Silhouette Clustering in Spark
Convert value depending on a type in SparkSQL via case matching of type
How to flatten nested lists in PySpark?
How to force Spark to evaluate DataFrame operations inline
Run Command on EMR Slaves?
How does Spark manage stages?
What row is used in dropDuplicates operator?
Create an empty array column of certain type in pyspark DataFrame
Ignoring non-spark config property: hive.exec.dynamic.partition.mode
How to CREATE TABLE USING delta with Spark 2.4.4?
Write and read raw byte arrays in Spark - using Sequence File SequenceFile
How to check if Spark RDD is in memory?
Can Spark code be run on cluster without spark-submit?
How to save a spark RDD in gzip format through pyspark
Parquet predicate pushdown
How to map variable names to features after pipeline
Find minimum for a timestamp through Spark groupBy dataframe
Config file to define JSON Schema Structure in PySpark

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!