I have some Scala code which runs with spark, but I simplified them:
// Not Serializable
class Config
object FileReader extends FileReader
class FileReader extends Serializable {
def read(config: Config): String = config.getClass.toString
}
object Task extends Task(FileReader)
class Task(fileReader: FileReader) extends Serializable {
def execute(config: Config): Unit = {
fileReader.read(config)
}
}
Not the Config
is not Serializable
.
I want to write some tests for them, and the instance of Task
need to be Serializable
because it may be serialized and send to spark workers.
I use this function to check if an object can be serialized or not:
def checkSerializable(obj: AnyRef, name: String) = {
println("### checking " + name + ": " + obj.getClass)
new ObjectOutputStream(new ByteArrayOutputStream()).writeObject(obj)
println(name + " ok")
}
First let's check if a normal Task
instance can be serialized:
val task = new Task(new FileReader)
checkSerializable(task, "no-mockito")
Outputs:
### checking no-mockito: class Task
no-mockito ok
Seems OK.
But I want to mock the FileReader
with Mockito, so my code will be:
val fileReader = Mockito.mock(classOf[FileReader])
val config = Mockito.mock(classOf[Config])
Mockito.when(fileReader.read(config)).thenReturn("mocked")
val task = new Task(fileReader)
checkSerializable(task, "with-mockito1")
It reports error that Config
is not serializable:
### checking with-mockito1: class Task
java.io.NotSerializableException: Config$$EnhancerByMockitoWithCGLIB$$c7dcb0a5
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1165)
at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1359)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1155)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
This is strange because config
is just some method parameters, not class fields!
I modified my code a little bit to make the config
serializable:
val fileReader = Mockito.mock(classOf[FileReader])
val config = Mockito.mock(classOf[Config], Mockito.withSettings().serializable())
Mockito.when(fileReader.read(config)).thenReturn("mocked")
val task = new Task(fileReader)
checkSerializable(task, "with-mockito2")
It still fails, with another NotSerializableException
:
### checking with-mockito2: class Task
java.io.NotSerializableException: org.mockito.internal.creation.DelegatingMethod
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1165)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
Not sure why config
instance will be included in task
when working with Mockito? And how to avoid it?
Demo project: https://github.com/freewind/mockito-serialization-issue, you can just clone it and run demo/MockitoDemo.scala
Update:
Another notable thing is: if I remove this line
fileReader.read(config)
from Task
, that means the Task
will be:
class Task(fileReader: FileReader) extends Serializable {
def execute(config: Config): Unit = {
// removed this line: fileReader.read(config)
}
}
no NotSerializableException
will be thrown anymore (I didn't change the testing code)
The error was triggered when I initialize a variable on the driver (master), but then tried to use it on one of the workers. When that happens, Spark Streaming will try to serialize the object to send it over to the worker, and fail if the object is not serializable. I solved the error by making the variable static.
This answer is in response to the question in the title, "Shouldn't Optional be Serializable?" The short answer is that the Java Lambda (JSR-335) expert group considered and rejected it.
Serializing an object means taking the data stored in an object and converting it to bytes (or a string). Suppose you want to write the data in an object to a JSON file. JSON files store strings. JSON has no understanding about the JVM or Scala objects.
You probably will need to make the FileReader mock serializable too try:
val fileReader = Mockito.mock(classOf[FileReader], Mockito.withSettings().serializable())
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With