I'm going through some Sapark projects in Scala, and I see that all of the objects are extending Serializable
.
It's like :
object someName extends Serializable {
//some code
}
I understand that serialization is often used for storing or communicating data structures, so that the data can be easily loaded into memory in the original form, from the serialized form. However, in this case the object
is more like a Java class. So, what's the point or advantage of extending Serializable
? When do you do this? Is it necessary to always do this?
Serialization is used for performance tuning on Apache Spark. All data that is sent over the network or written to the disk or persisted in the memory should be serialized. Serialization plays an important role in costly operations.
Serialization is the process of converting an object into a stream of bytes to store the object or transmit it to memory, a database, or a file. Its main purpose is to save the state of an object in order to be able to recreate it when needed.
Serializing an object means taking the data stored in an object and converting it to bytes (or a string). Suppose you want to write the data in an object to a JSON file. JSON files store strings. JSON has no understanding about the JVM or Scala objects.
What happens if you try to send non-serialized Object over network? When traversing a graph, an object may be encountered that does not support the Serializable interface. In this case the NotSerializableException will be thrown and will identify the class of the non-serializable object.
You only actually need it when objects are used other than calling methods directly on them. E.g. something like
val x = someName
rdd.map { ... x ... }
or
object SomeName extends Serializable, SomeTrait { ... }
def f(x: SomeTrait) = ...
rdd.map { ... f(SomeName) ... }
For "global" object
s not extending any classes/traits it's basically useless because no one ever does what the first snippet shows, but it doesn't hurt either. Note that Nil
and None
are examples of the second and they do extend Serializable
(even without Spark).
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With