Capturing unused fields while decoding a JSON object with circe

Question

Suppose I have a case class like the following, and I want to decode a JSON object into it, with all of the fields that haven't been used ending up in a special member for the leftovers:

import io.circe.Json

case class Foo(a: Int, b: String, leftovers: Json)

What's the best way to do this in Scala with circe?

(Note: I've seen questions like this a few times, so I'm Q-and-A-ing it for posterity.)

Answer 1

There are a couple of ways you could go about this. One fairly straightforward way would be to filter out the keys you've used after decoding:

import io.circe.{ Decoder, Json, JsonObject }

implicit val decodeFoo: Decoder[Foo] =
  Decoder.forProduct2[Int, String, (Int, String)]("a", "b")((_, _)).product(
    Decoder[JsonObject]
  ).map {
    case ((a, b), all) =>
      Foo(a, b, Json.fromJsonObject(all.remove("a").remove("b")))
  }

Which works as you'd expect:

scala> val doc = """{ "something": false, "a": 1, "b": "abc", "0": 0 }"""
doc: String = { "something": false, "a": 1, "b": "abc", "0": 0 }

scala> io.circe.jawn.decode[Foo](doc)
res0: Either[io.circe.Error,Foo] =
Right(Foo(1,abc,{
  "something" : false,
  "0" : 0
}))

The disadvantage of this approach is that you have to maintain code to remove the keys you've used separately from their use, which can be error-prone. Another approach is to use circe's state-monad-powered decoding tools:

import cats.data.StateT
import cats.instances.either._
import io.circe.{ ACursor, Decoder, Json }

implicit val decodeFoo: Decoder[Foo] = Decoder.fromState(
  for {
    a <- Decoder.state.decodeField[Int]("a")
    b <- Decoder.state.decodeField[String]("b")
    rest <- StateT.inspectF((_: ACursor).as[Json])
  } yield Foo(a, b, rest)
)

Which works the same way as the previous decoder (apart from some small differences in the errors you'll get if decoding fails):

scala> io.circe.jawn.decode[Foo](doc)
res1: Either[io.circe.Error,Foo] =
Right(Foo(1,abc,{
  "something" : false,
  "0" : 0
}))

This latter approach doesn't require you to change the used fields in multiple places, and it also has the advantage of looking a little more like any other decoder you'd write manually in circe.