Importing spark.implicits._ in scala

Question

I am trying to import spark.implicits._ Apparently, this is an object inside a class in scala. when i import it in a method like so:

def f() = {   val spark = SparkSession()....   import spark.implicits._ } 

It works fine, however i am writing a test class and i want to make this import available for all tests I have tried:

class SomeSpec extends FlatSpec with BeforeAndAfter {   var spark:SparkSession = _    //This won't compile   import spark.implicits._    before {     spark = SparkSession()....     //This won't either     import spark.implicits._   }    "a test" should "run" in {     //Even this won't compile (although it already looks bad here)     import spark.implicits._      //This was the only way i could make it work     val spark = this.spark     import spark.implicits._   } } 

Not only does this look bad, i don't want to do it for every test What is the "correct" way of doing it?

Answer 1

You can do something similar to what is done in the Spark testing suites. For example this would work (inspired by SQLTestData):

class SomeSpec extends FlatSpec with BeforeAndAfter { self =>    var spark: SparkSession = _    private object testImplicits extends SQLImplicits {     protected override def _sqlContext: SQLContext = self.spark.sqlContext   }   import testImplicits._    before {     spark = SparkSession.builder().master("local").getOrCreate()   }    "a test" should "run" in {     // implicits are working     val df = spark.sparkContext.parallelize(List(1,2,3)).toDF()   } } 

Alternatively you may use something like SharedSQLContext directly, which provides a testImplicits: SQLImplicits, i.e.:

class SomeSpec extends FlatSpec with SharedSQLContext {   import testImplicits._    // ...  } 

Answer 2

I think the GitHub code in SparkSession.scala file can give you a good hint:

      /**        * :: Experimental ::        * (Scala-specific) Implicit methods available in Scala for converting        * common Scala objects into [[DataFrame]]s.        *        * {{{        *   val sparkSession = SparkSession.builder.getOrCreate()        *   import sparkSession.implicits._        * }}}        *        * @since 2.0.0        */       @Experimental       object implicits extends SQLImplicits with Serializable {         protected override def _sqlContext: SQLContext = SparkSession.this.sqlContext       } 

here "spark" in "spark.implicits._" is just the sparkSession object we created.

Here is another reference!