Spark : Scala mocking, Task not serializable

Question

I am trying to use mockito for unit testing some scala code. I want to run spark locally, i.e. in my IntelliJ IDE. Here is a sample

class MyScalaSparkTests extends FunSuite with BeforeAndAfter with MockitoSugar with java.io.Serializable{

  val configuration:SparkConf  = new SparkConf()
    .setAppName("Your Application Name")
    .setMaster("local");
  val sc = new SparkContext(configuration);
  lazy val testSess = SparkSession.builder.appName("local_test").getOrCreate()
  test ("test service") {
    import testSess.implicits._
    // (1) init
    val testObject = spy(new MyScalaClass(<some args>))
    val testDf = testSess.emptyDataset[MyCaseClass1].toDF()
    testDf.union(Seq(MyCaseClass(<some args>)).toDF())
    testObject.testDataFrame = testDf
    val testSource = testSess.emptyDataset[MyCaseClass2].toDF()
    testSource.union(Seq(MyCaseClass2(<some args>)).toDF())
    testObject.setSourceDf(testSource)
    val testRes = testObject.someMethod()

    val r = testRes.take(1)
    println(r)

  }

}

so basically, here is what I am trying to do

MyScalaClass has someMethod() which compares data between two data frames called testDataFrame and testSource. It then returns another data frame which has the results. Now, in my unit test, I am spying on MyScalaClass to create testObject. Then I create testDataFrame and testSource and assign them to testObject. Finally, I call testObject.someMethod().

Now in the debugger, at this line

val r = testRes.take(1)

I see that testRes is a Dataset hence something is being returned by the method. But when I try to take something from it in order to verify the results I get

Task not serializable
org.apache.spark.SparkException: Task not serializable

and further down the stacktrace

Caused by: java.io.NotSerializableException: org.mockito.internal.creation.DelegatingMethod
Serialization stack:
    - object not serializable (class: org.mockito.internal.creation.DelegatingMethod, value: org.mockito.internal.creation.DelegatingMethod@a97f2bff)
    - field (class: org.mockito.internal.invocation.InterceptedInvocation, name: mockitoMethod, type: interface org.mockito.internal.invocation.MockitoMethod)
    - object (class org.mockito.internal.invocation.InterceptedInvocation, bSV2PartValidator.toString();)
    - field (class: org.mockito.internal.invocation.InvocationMatcher, name: invocation, type: interface org.mockito.invocation.Invocation)
    - object (class org.mockito.internal.invocation.InvocationMatcher, bSV2PartValidator.toString();)
    - field (class: org.mockito.internal.stubbing.InvocationContainerImpl, name: invocationForStubbing, type: interface org.mockito.invocation.MatchableInvocation)
    - object (class org.mockito.internal.stubbing.InvocationContainerImpl, invocationForStubbing: bSV2PartValidator.toString();)
    - field (class: org.mockito.internal.handler.MockHandlerImpl, name: invocationContainer, type: class org.mockito.internal.stubbing.InvocationContainerImpl)
    - object (class org.mockito.internal.handler.MockHandlerImpl, org.mockito.internal.handler.MockHandlerImpl@47c019d7)
    - field (class: org.mockito.internal.handler.NullResultGuardian, name: delegate, type: interface org.mockito.invocation.MockHandler)
    - object (class org.mockito.internal.handler.NullResultGuardian, org.mockito.internal.handler.NullResultGuardian@7222e168)
    - field (class: org.mockito.internal.handler.InvocationNotifierHandler, name: mockHandler, type: interface org.mockito.invocation.MockHandler)
    - object (class org.mockito.internal.handler.InvocationNotifierHandler, org.mockito.internal.handler.InvocationNotifierHandler@1e4f8430)
    - field (class: org.mockito.internal.creation.bytebuddy.MockMethodInterceptor, name: handler, type: interface org.mockito.invocation.MockHandler)
    - object (class org.mockito.internal.creation.bytebuddy.MockMethodInterceptor, org.mockito.internal.creation.bytebuddy.MockMethodInterceptor@34d08905)
    - field (class: com.walmart.labs.search.signals.validators.BSV2PartValidator$MockitoMock$213785213, name: mockitoInterceptor, type: class org.mockito.internal.creation.bytebuddy.MockMethodInterceptor)
    - object (class com.walmart.labs.search.signals.validators.BSV2PartValidator$MockitoMock$213785213, com.walmart.labs.search.signals.validators.BSV2PartValidator$MockitoMock$213785213@7f289126)
    - field (class: com.walmart.labs.search.signals.validators.BSV2PartValidator$$anonfun$1, name: $outer, type: class com.walmart.labs.search.signals.validators.BSV2PartValidator)
    - object (class com.walmart.labs.search.signals.validators.BSV2PartValidator$$anonfun$1, <function1>)
    - element of array (index: 1)
    - array (class [Ljava.lang.Object;, size 7)
    - field (class: org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8, name: references$1, type: class [Ljava.lang.Object;)
    - object (class org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8, <function2>)
    at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
    at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:46)
    at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:295)
    ... 78 more

What am i doing wrong? Is it even possible to spy on or mock spark behavior in IDE?

Answer 1

Mocks are not serialisable by default, as it's usually a code smell in unit testing

You can try enabling serialisation by creating the mock like mock[MyType](Mockito.withSettings().serializable()) and see what happens when spark tries to use it.

BTW, I recommend you to use mockito-scala instead of the traditional mockito as it may save you some other problems