I work with Spark often, and it would save me a lot of time if the compiler could ensure that a type is serializable.
Perhaps with a type class?
def foo[T: IsSerializable](t: T) = {
// do stuff requiring T to be serializable
}
It's not enough to constrain T <: Serializable. It could still fail at runtime. Unit tests are a good substitute, but you can still forget them, especially when working with big teams.
I think this is probably impossible to do at compile time without the types being sealed.
Yes, it is possible, but not in the way that you're hoping. Your type class IsSerializable
could provide a mechanism to convert your T
to a value of a type which is guaranteed to be Serializable
and back again,
trait IsSerializable[T] {
def toSerializable(t: T): String
def fromSerializable(s: String): Option[T]
}
But, of course, this is just an alternative type class based serialization mechanism in it's own right, making the use of JVM serialization redundant.
Your best course of action would be to lobby Spark to support type class based serialization directly.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With