Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there a way to ensure a type is Serializable at compile time

I work with Spark often, and it would save me a lot of time if the compiler could ensure that a type is serializable.

Perhaps with a type class?

def foo[T: IsSerializable](t: T) = {
  // do stuff requiring T to be serializable
}

It's not enough to constrain T <: Serializable. It could still fail at runtime. Unit tests are a good substitute, but you can still forget them, especially when working with big teams.

I think this is probably impossible to do at compile time without the types being sealed.

like image 865
Upio Avatar asked Sep 02 '16 07:09

Upio


1 Answers

Yes, it is possible, but not in the way that you're hoping. Your type class IsSerializable could provide a mechanism to convert your T to a value of a type which is guaranteed to be Serializable and back again,

trait IsSerializable[T] {
  def toSerializable(t: T): String
  def fromSerializable(s: String): Option[T]
}

But, of course, this is just an alternative type class based serialization mechanism in it's own right, making the use of JVM serialization redundant.

Your best course of action would be to lobby Spark to support type class based serialization directly.

like image 84
Miles Sabin Avatar answered Nov 15 '22 03:11

Miles Sabin