How to parse JSON with variable keys in Scala Play?

Question

Happy new years, first of all!

I'm having some problem parsing JSON in Play, the format I'm dealing with is as follows:

JSON Response:

 ...
"image":{  
    "large":{  
      "path":"http://url.jpg",
      "width":300,
      "height":200
    },
    "medium":{  
      "path":"http://url.jpg",
      "width":200,
      "height":133
    },
    ...
 }
...

I'm stuck on the field with the sizes. They are obviously variables, and I'm not sure how to write a formatter for this? The JSON is coming from an external service.

So far I have

final case class Foo(
  ..
  ..
  image: Option[Image])


final case class Image(size: List[Size])

final case class Size(path: String, width: Int, height: Int)

For the formatting I just did Json.reads[x] for all the classes. However I'm pretty sure that the variable for size is throwing off the formatting because it's not able to create an Image object from the JSON coming in.

Answer 1

Update 2016-07-28

The solution described below breaks Referential Transparency because of the use of the return keyword and is not something I would recommend today. Nevertheless, I am not leaving it as it for historical reasons.

Intro

The issue here is that you need to find someplace to save the key for each Size object in the Image object. There are two ways to do this, one is to save it in the Size object itself. This makes sense because the name is intimately related to the Size object, and it is convenient to store it there. So lets explore that solution first.

A quick note on Symmetry

Before we dive into any solutions, let me first introduce the concept of symmetry. This the idea that when you read any Json value, you can use your Scala model representation to go back to exactly the same Json value.

Symmetry when dealing with marshalled data is not strictly required, indeed sometimes it is either not possible, or enforcing it would be too costly without any real gain. But usually it is fairly easy to achieve and it makes working with the serialization implementation much nicer. In many cases it is required as well.

Save name in Size

import play.api.libs.json.Format
import play.api.libs.json.JsPath
import play.api.libs.json.Reads
import play.api.libs.json.JsValue
import play.api.libs.json.JsResult
import play.api.libs.json.JsSuccess
import play.api.libs.json.JsError
import play.api.libs.json.JsObject
import play.api.libs.json.Json

final case class Foo(images: Option[Image])

object Foo {
  implicit val fooFormat: Format[Foo] = Json.format[Foo]
}

final case class Image(sizes: Seq[Size])

object Image {

  implicit val imagesFormat: Format[Image] =
    new Format[Image] {

      /** @inheritdoc */
      override def reads(json: JsValue): JsResult[Image] = json match {
        case j: JsObject => {
          JsSuccess(Image(j.fields.map{
            case (name, size: JsObject) =>
              if(size.keys.size == 3){
                val valueMap = size.value
                valueMap.get("path").flatMap(_.asOpt[String]).flatMap(
                  p=> valueMap.get("height").flatMap(_.asOpt[Int]).flatMap(
                    h => valueMap.get("width").flatMap(_.asOpt[Int]).flatMap(
                      w => Some(Size(name, p, h, w))
                    ))) match {
                  case Some(value) => value
                  case None => return JsError("Invalid input")
                }
              } else {
                  return JsError("Invalid keys on object")
              }
            case _ =>
              return JsError("Invalid JSON Type")
          }))
        }
        case _ => JsError("Invalid Image")
      }

      /** @inheritdoc */
      override def writes(o: Image): JsValue = {
        JsObject(o.sizes.map((s: Size) =>
          (s.name ->
            Json.obj(
              ("path" -> s.path),
              ("height" -> s.height),
              ("width" -> s.width)))))
      }
    }

}

final case class Size(name: String, path: String, height: Int, width: Int)

In this solution Size does not have any Json serialization or deserialization directly, rather it comes as a product of the Image object. This is because, in order to have symmetric serialization of your Image object you need to keep not only the parameters of the Size object, path, height, and width, but also the name of the Size as specified as the keys on the Image object. If you don't store this you can't go back and forth freely.

So this works as we can see below,

scala> import play.api.libs.json.Json
import play.api.libs.json.Json

scala> Json.parse("""
     | {  
     |     "large":{  
     |       "path":"http://url.jpg",
     |       "width":300,
     |       "height":200
     |     },
     |     "medium":{  
     |       "path":"http://url.jpg",
     |       "width":200,
     |       "height":133
     |     }
     | }""")
res0: play.api.libs.json.JsValue = {"large":{"path":"http://url.jpg","width":300,"height":200},"medium":{"path":"http://url.jpg","width":200,"height":133}}

scala> res0.validate[Image]
res1: play.api.libs.json.JsResult[Image] = JsSuccess(Image(ListBuffer(Size(large,http://url.jpg,200,300), Size(medium,http://url.jpg,133,200))),)

scala> 

And very importantly it is both safe and symmetric

scala> Json.toJson(res0.validate[Image].get)
res4: play.api.libs.json.JsValue = {"large":{"path":"http://url.jpg","height":200,"width":300},"medium":{"path":"http://url.jpg","height":133,"width":200}}

scala> 

A quick note on Safety

In production code, you never, never, never want to use the .as[T] method on a JsValue. This is because if the data is not what you expected, it blows up without any meaningful error handling. If you must, use .asOpt[T], but a much better choice in general is .validate[T], as this will produce some form of error on failure that you can log and then report back to the user.

Probably a Better Solution

Now, probably a better way to do accomplish this would be to change the Image case class declaration to the following

final case class Image(s: Seq[(String, Size)])

and then keep Size as you originally had it,

final case class Size(path: String, height: Int, width: Int)

Then you merely need to do the following to be safe and symmetric.

If we do this then the implementation becomes much nicer, while still being safe and symmetric.

import play.api.libs.json.Format
import play.api.libs.json.JsPath
import play.api.libs.json.Reads
import play.api.libs.json.JsValue
import play.api.libs.json.JsResult
import play.api.libs.json.JsSuccess
import play.api.libs.json.JsError
import play.api.libs.json.JsObject
import play.api.libs.json.Json

final case class Foo(images: Option[Image])

object Foo {
  implicit val fooFormat: Format[Foo] = Json.format[Foo]
}

final case class Image(sizes: Seq[(String, Size)])

object Image {

  implicit val imagesFormat: Format[Image] =
    new Format[Image] {

      /** @inheritdoc */
      override def reads(json: JsValue): JsResult[Image] = json match {
        case j: JsObject =>
          JsSuccess(Image(j.fields.map{
            case (name, size) =>
              size.validate[Size] match {
                case JsSuccess(validSize, _) => (name, validSize)
                case e: JsError => return e
              }
          }))
        case _ =>
          JsError("Invalid JSON type")
      }

      /** @inheritdoc */
      override def writes(o: Image): JsValue = Json.toJson(o.sizes.toMap)
    }
}

final case class Size(path: String, height: Int, width: Int)

object Size {
  implicit val sizeFormat: Format[Size] = Json.format[Size]
}

Still works as before

scala> Json.parse("""
     | {
     | "large":{  
     |       "path":"http://url.jpg",
     |       "width":300,
     |       "height":200
     |     },
     |     "medium":{  
     |       "path":"http://url.jpg",
     |       "width":200,
     |       "height":133}}""")
res1: play.api.libs.json.JsValue = {"large":{"path":"http://url.jpg","width":300,"height":200},"medium":{"path":"http://url.jpg","width":200,"height":133}}

scala> res1.validate[Image]
res2: play.api.libs.json.JsResult[Image] = JsSuccess(Image(ListBuffer((large,Size(http://url.jpg,200,300)), (medium,Size(http://url.jpg,133,200)))),)

scala> Json.toJson(res1.validate[Image].get)
res3: play.api.libs.json.JsValue = {"large":{"path":"http://url.jpg","height":200,"width":300},"medium":{"path":"http://url.jpg","height":133,"width":200}}

But with the benefit that Size is now reflective of real Json, that is you can serialize and deserialize just Size values. Which makes it both easier to work with and think about.

TL;DR Commentary on reads in the First Example

Although I would argue that the first solution is somewhat inferior to the second solution, we did use some interesting idioms in the first implementation of reads that are very useful, in a more general sense, but often not well understood. So I wanted to take the time to go through them in more detail for those who are interested. If you already understand the idioms in use, or you just don't care, feel free to skip this discussion.

flatMap chaining

When we attempt to get the values we need out of valueMap, at any an all steps things can go wrong. We would like to handle these cases reasonably without catastrophic exceptions being thrown.

To accomplish this we use the Option value and common flatMap function to chain our computation. There are really two steps we do for each desired value, get the value out of valueMap and we force it to the proper type using the asOpt[T] function. Now the nice thing is that both valueMap.get(s: String) and jsValue.asOpt[T] both return Option values. This means that we can use flatMap to build our final result. flatMap has the nice property that if any of the steps in the flatMap chain fail, i.e. return None, then all other steps are not run and the final result is returned as None.

This idiom is part of general Monadic programming that is common to functional languages, especially Haskell and Scala. In Scala it is not often referred to as Monadic because when the concept was introduced in Haskell it was often explained poorly leading to many people disliking it, despite it in fact being very useful. Due to this, people are often afraid to use the "M word" with respect to Scala.

Functional Short Circuiting

The other idiom that is used in reads, in both versions, is short circuiting a function call by using the return keyword in scala.

As you probably know, use of the return keyword is often discouraged in Scala, as the final value of any function is automatically made into the return value for the function. There is however one very useful time to use the return keyword, that is when you are calling a function that represents a repeated call over something, such as the map function. If you hit some terminal condition on one of the inputs you can use the return keyword to stop the execution of the map call on the remaining elements. It is somewhat analogous to using break in a for loop in languages like Java.

In our case, we wanted to ensure certain things about the elements in the Json, like that it had the correct keys and types, and if at any point any of our assumptions were incorrect, we wanted to return the proper error information. Now we could just map over the fields in the Json, and then inspect the result after the map operation has completed, but consider if someone had sent us very large Json with thousands of keys that did not have the structure we wanted. We would have to apply our function to all of the values even if we knew we had an error after only the first application. Using return we can end the map application as soon we know about an error, without having to spend time apply the map application across the rest of the elements when the result is already known.

Anyway, I hope that little bit of pedantic explanation is helpful!