Scala pickling case class versioning

Question

I would like to be able to use scala pickling in order to store binary representation of a case class.

I would like to know if there is a way to manage versioning of case class (the way protocol buffer allows to do)

---

Here is my example

I make a program at a certain date, with the following case class

case class MessageTest(a:String,b:String) 

Then I serialize an instance of this class

import scala.pickling._
import binary._
val bytes=MessageTest("1","2").pickle

And then I store the result into a file

---

Later on, I might now have to make evolution on my case class, to add a new optional field

case class MessageTest (a:String,b:String,c:Option[String]=None)

I would like to be able to reuse the data that I stored previously in my file , to deserialize it and be able to recover an instance of a case class (with default value for the new parameter)

But when I use the following code

import scala.pickling._
import binary._
val messageback=bytes.unpickle[MessageTest]

I got the following error :

java.lang.ArrayIndexOutOfBoundsException: 26 at scala.pickling.binary.BinaryPickleReader$$anonfun$2.apply(BinaryPickleFormat.scala:446) at scala.pickling.binary.BinaryPickleReader$$anonfun$2.apply(BinaryPickleFormat.scala:434) at scala.pickling.PickleTools$class.withHints(Tools.scala:498) at scala.pickling.binary.BinaryPickleReader.withHints(BinaryPickleFormat.scala:425) at scala.pickling.binary.BinaryPickleReader.beginEntryNoTagDebug(BinaryPickleFormat.scala:434) at scala.pickling.binary.BinaryPickleReader.beginEntryNoTag(BinaryPickleFormat.scala:431)

---

Did I do something wrong ?

Is there an existing way to make my scenario work ?

Regards

Answer 1

Well the problem is that you are trying to deserialize back to a different object than what you serialized to.

Consider this. The first object

scala> case class MessageTest(a: String, b:String)
defined class MessageTest

scala> val bytes = MessageTest("a", "b").pickle
bytes: pickling.binary.pickleFormat.PickleType = BinaryPickle([0,0,0,81,36,108,105,110,101,53,49,46,36,114,101,97,100,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,77,101,115,115,97,103,101,84,101,115,116,0,0,0,1,97,0,0,0,1,98])

Now with the changed case object...

scala> case class MessageTest(a: String, b: String, c: Option[String] = None)
defined class MessageTest

scala> val bytes = MessageTest("a", "b").pickle
bytes: pickling.binary.pickleFormat.PickleType = BinaryPickle([0,0,0,81,36,108,105,110,101,53,51,46,36,114,101,97,100,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,36,105,119,46,77,101,115,115,97,103,101,84,101,115,116,0,0,0,1,97,0,0,0,1,98,0,0,0,15,115,99,97,108,97,46,78,111,110,101,46,116,121,112,101])

There is no way the library can know what you mean in this case because it's just expecting signatures to match up.

https://github.com/scala/pickling/issues/39 but you can at least go one way with it. as demonstrated here.

import scala.pickling._
import scala.pickling.Defaults._
import scala.pickling.binary._

case class LegacyMessage(a: String, b: String)
case class Message(a: String, b: String, c: Option[String] = None)

implicit val legacyUnpickler = Unpickler.generate[LegacyMessage]
implicit val messageUnpickler = Unpickler.generate[Message]

val legacyBytes = LegacyMessage("a", "b")
val msgBytes = Message("a", "b", None)

val pickledBytes = msgBytes.pickle
val pickledLegacy = legacyBytes.pickle

// New Message can Serialize back to Legacy Messages
val newToOld = pickledBytes.unpickle[LegacyMessage]

// Old Messages can not serialize up to the new message schema
// println(pickledLegacy.unpickle[Message])

val old = pickledLegacy.unpickle[LegacyMessage]

if(newToOld == old){
  println(true)
}

Hopefully this helps a little.