How to shapeless case classes with attributes and typeclasses?

Question

I am currently implementing a library to serialize and deserialize to and from XML-RPC messages. It's almost done but now I am trying to remove the boilerplate of my current asProduct method using Shapeless. My current code:

trait Serializer[T] {
  def serialize(value: T): NodeSeq
} 

trait Deserializer[T] {
  type Deserialized[T] = Validation[AnyErrors, T]
  type AnyErrors = NonEmptyList[AnyError]
  def deserialize(from: NodeSeq): Deserialized[T]
}

trait Datatype[T] extends Serializer[T] with Deserializer[T]

// Example of asProduct, there are 20 more methods like this, from arity 1 to 22
def asProduct2[S, T1: Datatype, T2: Datatype](apply: (T1, T2) => S)(unapply: S => Product2[T1, T2]) = new Datatype[S] {
  override def serialize(value: S): NodeSeq = {
    val params = unapply(value)
    val b = toXmlrpc(params._1) ++ toXmlrpc(params._2)
    b.theSeq
  }

  // Using scalaz
  override def deserialize(from: NodeSeq): Deserialized[S] = (
      fromXmlrpc[T1](from(0)) |@| fromXmlrpc[T2](from(1))
    ) {apply}
}

My goal is to allow the user of my library to serialize/deserialize case classes without force him to write boilerplate code. Currently, you have to declare the case class and an implicit val using the aforementioned asProduct method to have a Datatype instance in context. This implicit is used in the following code:

def toXmlrpc[T](datatype: T)(implicit serializer: Serializer[T]): NodeSeq =
  serializer.serialize(datatype)

def fromXmlrpc[T](value: NodeSeq)(implicit deserializer: Deserializer[T]): Deserialized[T] =
  deserializer.deserialize(value)

This is the classic strategy of serializing and deserializing using type classes.

At this moment, I have grasped how to convert from case classes to HList via Generic or LabelledGeneric. The problem is once I have this conversion done how I can call the methods fromXmlrpc and toXmlrpc as in the asProduct2 example. I don't have any information about the types of the attributes in the case class and, therefore, the compiler cannot find any implicit that satisfy fromXmlrpc and toXmlrpc. I need a way to constrain that all the elements of a HList have an implicit Datatype in context.

As I am a beginner with Shapeless, I would like to know what's the best way of getting this functionality. I have some insights but I definitely have no idea of how to get it done using Shapeless. The ideal would be to have a way to get the type from a given attribute of the case class and pass this type explicitly to fromXmlrpc and toXmlrpc. I imagine that this is not how it can be done.

Answer 1

First, you need to write generic serializers for HList. That is, you need to specify how to serialize H :: T and HNil:

implicit def hconsDatatype[H, T <: HList](implicit hd: Datatype[H],
                                          td: Datatype[T]): Datatype[H :: T] =
  new Datatype[H :: T] {
    override def serialize(value: H :: T): NodeSeq = value match {
      case h :: t =>
        val sh = hd.serialize(h)
        val st = td.serialize(t)
        (sh ++ st).theSeq
    }

    override def deserialize(from: NodeSeq): Deserialized[H :: T] =
      (hd.deserialize(from.head) |@| td.deserialize(from.tail)) {
        (h, t) => h :: t
      }
  }

implicit val hnilDatatype: Datatype[HNil] =
  new Datatype[HNil] {
    override def serialize(value: HNil): NodeSeq = NodeSeq()
    override def deserialize(from: NodeSeq): Deserialized[HNil] =
      Success(HNil)
  }

Then you can define a generic serializer for any type which can be deconstructed via Generic:

implicit def genericDatatype[T, R](implicit gen: Generic.Aux[T, R],
                                   rd: Lazy[Datatype[R]]): Datatype[T] =
  new Datatype[T] {
    override def serialize(value: T): NodeSeq =
      rd.value.serialize(gen.to(value))

    override def deserialize(from: NodeSeq): Deserialized[T] =
      rd.value.deserialize(from).map(rd.from)
  }

Note that I had to use Lazy because otherwise this code would break the implicit resolution process if you have nested case classes. If you get "diverging implicit expansion" errors, you could try adding Lazy to implicit parameters in hconsDatatype and hnilDatatype as well.

This works because Generic.Aux[T, R] links the arbitrary product-like type T and a HList type R. For example, for this case class

case class A(x: Int, y: String)

shapeless will generate a Generic instance of type

Generic.Aux[A, Int :: String :: HNil]

Consequently, you can delegate the serialization to recursively defined Datatypes for HList, converting the data to HList with Generic first. Deserialization works similarly but in reverse - first the serialized form is read to HList and then this HList is converted to the actual data with Generic.

It is possible that I made several mistakes in the usage of NodeSeq API above but I guess it conveys the general idea.

If you want to use LabelledGeneric, the code would become slightly more complex, and even more so if you want to handle sealed trait hierarchies which are represented with Coproducts.

I'm using shapeless to provide generic serialization mechanism in my library, picopickle. I'm not aware of any other library which does this with shapeless. You can try and find some examples how shapeless could be used in this library, but the code there is somewhat complex. There is also an example among shapeless examples, namely S-expressions.

Answer 2

Vladimir's answer is great and should be the accepted one, but it's also possible to do this a little more nicely with Shapeless's TypeClass machinery. Given the following setup:

import scala.xml.NodeSeq
import scalaz._, Scalaz._

trait Serializer[T] {
  def serialize(value: T): NodeSeq
} 

trait Deserializer[T] {
  type Deserialized[T] = Validation[AnyErrors, T]
  type AnyError = Throwable
  type AnyErrors = NonEmptyList[AnyError]
  def deserialize(from: NodeSeq): Deserialized[T]
}

trait Datatype[T] extends Serializer[T] with Deserializer[T]

We can write this:

import shapeless._

object Datatype extends ProductTypeClassCompanion[Datatype] {
  object typeClass extends ProductTypeClass[Datatype] {
    def emptyProduct: Datatype[HNil] = new Datatype[HNil] {
      def serialize(value: HNil): NodeSeq = Nil
      def deserialize(from: NodeSeq): Deserialized[HNil] = HNil.successNel
    }

    def product[H, T <: HList](
      dh: Datatype[H],
      dt: Datatype[T]
    ): Datatype[H :: T] = new Datatype[H :: T] {
      def serialize(value: H :: T): NodeSeq =
        dh.serialize(value.head) ++ dt.serialize(value.tail)

      def deserialize(from: NodeSeq): Deserialized[H :: T] =
       (dh.deserialize(from.head) |@| dt.deserialize(from.tail))(_ :: _)
    }

    def project[F, G](
      instance: => Datatype[G],
      to: F => G,
      from: G => F
    ): Datatype[F] = new Datatype[F] {
      def serialize(value: F): NodeSeq = instance.serialize(to(value))
      def deserialize(nodes: NodeSeq): Deserialized[F] =
        instance.deserialize(nodes).map(from)
    }
  }
}

Be sure to define these all together so they'll be properly companioned.

Then if we have a case class:

case class Foo(bar: String, baz: String)

And instances for the types of the case class members (in this case just String):

implicit object DatatypeString extends Datatype[String] {
  def serialize(value: String) = <s>{value}</s>
  def deserialize(from: NodeSeq) = from match {
    case <s>{value}</s> => value.text.successNel
    case _ => new RuntimeException("Bad string XML").failureNel
  }
}

We automatically get a derived instance for Foo:

scala> case class Foo(bar: String, baz: String)
defined class Foo

scala> val fooDatatype = implicitly[Datatype[Foo]]
fooDatatype: Datatype[Foo] = Datatype$typeClass$$anon$3@2e84026b

scala> val xml = fooDatatype.serialize(Foo("AAA", "zzz"))
xml: scala.xml.NodeSeq = NodeSeq(<s>AAA</s>, <s>zzz</s>)

scala> fooDatatype.deserialize(xml)
res1: fooDatatype.Deserialized[Foo] = Success(Foo(AAA,zzz))

This works about the same as Vladimir's solution, but it lets Shapeless abstract some of the boring boilerplate of type class instance derivation so you don't have to get your hands dirty with Generic.