Flattening nested JSON objects with Circe

Question

Suppose I have a JSON object like this:

{
   "foo": true,
   "bar": {
      "baz": 1,
      "qux": {
        "msg": "hello world",
        "wow": [null]
      }
   }
}

And I want to flatten it recursively to a single layer, with the keys merged with an underscore:

{
   "foo": true,
   "bar_baz": 1,
   "baz_qux_msg": "hello world",
   "baz_qux_wow": [null]
}

How can I do this with Circe?

(Note: this is another FAQ from the Circe Gitter channel.)

Answer 1

You can do this without too much pain in Circe with a recursive method:

import io.circe.Json

def flatten(combineKeys: (String, String) => String)(value: Json): Json = {
  def flattenToFields(value: Json): Option[Iterable[(String, Json)]] =
    value.asObject.map(
      _.toIterable.flatMap {
        case (k, v) => flattenToFields(v) match {
          case None => List(k -> v)
          case Some(fields) => fields.map {
            case (innerK, innerV) => combineKeys(k, innerK) -> innerV
          }
        }
      }
    )

  flattenToFields(value).fold(value)(Json.fromFields)
}

Here our internal flattenToFields method takes each JSON value and either returns None if it's a non-JSON object value, as a signal that that field doesn't need flattening, or a Some containing a sequence of flattened fields in the case of a JSON object.

If we have a JSON value like this:

val Right(doc) = io.circe.jawn.parse("""{
   "foo": true,
   "bar": {
      "baz": 1,
      "qux": {
        "msg": "hello world",
        "wow": [null]
      }
   }
}""")

We can verify that flatten does what we want like this:

scala> flatten(_ + "_" + _)(doc)
res1: io.circe.Json =
{
  "foo" : true,
  "bar_baz" : 1,
  "bar_qux_msg" : "hello world",
  "bar_qux_wow" : [
    null
  ]
}

Note that flattenToFields is not tail recursive, and will overflow the stack for deeply-nested JSON objects, but probably not until you're several thousand levels deep, so it's unlikely to be an issue in practice. You could make it tail recursive without too much trouble, but at the expense of additional overhead for the common cases where you only have a few layers of nesting.