I have some Scala code which runs with spark, but I simplified them:
// Not Serializable
class Config
object FileReader extends FileReader
class FileReader extends Serializable {
def read(config: Config): String = config.getClass.toString
}
object Task extends Task(FileReader)
class Task(fileReader: FileReader) extends Serializable {
def execute(config: Config): Unit = {
fileReader.read(config)
}
}
Not the Config is not Serializable.
I want to write some tests for them, and the instance of Task need to be Serializable because it may be serialized and send to spark workers.
I use this function to check if an object can be serialized or not:
def checkSerializable(obj: AnyRef, name: String) = {
println("### checking " + name + ": " + obj.getClass)
new ObjectOutputStream(new ByteArrayOutputStream()).writeObject(obj)
println(name + " ok")
}
First let's check if a normal Task instance can be serialized:
val task = new Task(new FileReader)
checkSerializable(task, "no-mockito")
Outputs:
### checking no-mockito: class Task
no-mockito ok
Seems OK.
But I want to mock the FileReader with Mockito, so my code will be:
val fileReader = Mockito.mock(classOf[FileReader])
val config = Mockito.mock(classOf[Config])
Mockito.when(fileReader.read(config)).thenReturn("mocked")
val task = new Task(fileReader)
checkSerializable(task, "with-mockito1")
It reports error that Config is not serializable:
### checking with-mockito1: class Task
java.io.NotSerializableException: Config$$EnhancerByMockitoWithCGLIB$$c7dcb0a5
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1165)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1359)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1155)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
This is strange because config is just some method parameters, not class fields!
I modified my code a little bit to make the config serializable:
val fileReader = Mockito.mock(classOf[FileReader])
val config = Mockito.mock(classOf[Config], Mockito.withSettings().serializable())
Mockito.when(fileReader.read(config)).thenReturn("mocked")
val task = new Task(fileReader)
checkSerializable(task, "with-mockito2")
It still fails, with another NotSerializableException:
### checking with-mockito2: class Task
java.io.NotSerializableException: org.mockito.internal.creation.DelegatingMethod
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1165)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
Not sure why config instance will be included in task when working with Mockito? And how to avoid it?
Demo project: https://github.com/freewind/mockito-serialization-issue, you can just clone it and run demo/MockitoDemo.scala
Update:
Another notable thing is: if I remove this line
fileReader.read(config)
from Task, that means the Task will be:
class Task(fileReader: FileReader) extends Serializable {
def execute(config: Config): Unit = {
// removed this line: fileReader.read(config)
}
}
no NotSerializableException will be thrown anymore (I didn't change the testing code)
The error was triggered when I initialize a variable on the driver (master), but then tried to use it on one of the workers. When that happens, Spark Streaming will try to serialize the object to send it over to the worker, and fail if the object is not serializable. I solved the error by making the variable static.
This answer is in response to the question in the title, "Shouldn't Optional be Serializable?" The short answer is that the Java Lambda (JSR-335) expert group considered and rejected it.
Serializing an object means taking the data stored in an object and converting it to bytes (or a string). Suppose you want to write the data in an object to a JSON file. JSON files store strings. JSON has no understanding about the JVM or Scala objects.
You probably will need to make the FileReader mock serializable too try:
val fileReader = Mockito.mock(classOf[FileReader], Mockito.withSettings().serializable())
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With