Spark : Error Not found value SC

Question

I have just started with Spark. I have CDH5 Installed with Spark . However when I try to use sparkcontext it gives Error as below

<console>:17: error: not found: value sc
       val distdata = sc.parallelize(data)

I have researched about this and found error: not found: value sc

and tried to start spark context with ./spark-shell . It gives error No such File or Directory

Answer 1

You can either start spark-shell starting with ./ if you're in its exact directory or path/to/spark-shell if you're elsewhere.

Also, if you're running a script with spark-submit, you need to initialize sc as SparkContext first:

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

val conf = new SparkConf().setAppName("Simple Application")
val sc = new SparkContext(conf)

Answer 2

There is another stackoverflow post that answers this question by getting sc(spark context) from spark session. I do it this way:

val spark = SparkSession.builder().appName("app_name").enableHiveSupport().getOrCreate()

val sc = spark.sparkContext

original answer here: Retrieve SparkContext from SparkSession