What `JObject(rec) <- someJArray` means inside for-comprehension

Question

I'm learning Json4s library.

I have a json fragment like this:

{
    "records":[
        {
            "name":"John Derp",
            "address":"Jem Street 21"
        },
        {
            "name":"Scala Jo",
            "address":"in my sweet dream"
        }
    ]
}

And, I have Scala code, which converts a json string into a List of Maps, like this:

import org.json4s._
import org.json4s.JsonAST._
import org.json4s.native.JsonParser

  val json = JsonParser.parse( """{"records":[{"name":"John Derp","address":"Jem Street 21"},{"name":"Scala Jo","address":"in my sweet dream"}]}""")

  val records: List[Map[String, Any]] = for {
    JObject(rec) <- json \ "records"
    JField("name", JString(name)) <- rec
    JField("address", JString(address)) <- rec
  } yield Map("name" -> name, "address" -> address)

  println(records)

The output of records to screen gives this:

List(Map(name -> John Derp, address -> Jem Street 21), Map(name -> Scala Jo, address -> in my sweet dream))

I want to understand what the lines inside the for loop mean. For example, what is the meaning of this line:

JObject(rec) <- json \ "records"

I understand that the json \ "records" produces a JArray object, but why is it fetched as JObject(rec) at left of <-? What is the meaning of the JObject(rec) syntax? Where does the rec variable come from? Does JObject(rec) mean instantiating a new JObject class from rec input?

BTW, I have a Java programming background, so it would also be helpful if you can show me the Java equivalent code for the loop above.

Answer 1

You have the following types hierarchy:

  sealed abstract class JValue {
    def \(nameToFind: String): JValue = ???
    def filter(p: (JValue) => Boolean): List[JValue] = ???
  }

  case class JObject(val obj: List[JField]) extends JValue
  case class JField(val name: String, val value: JValue) extends JValue
  case class JString(val s: String) extends JValue
  case class JArray(val arr: List[JValue]) extends JValue {
    override def filter(p: (JValue) => Boolean): List[JValue] = 
      arr.filter(p)
  }

Your JSON parser returns following object:

  object JsonParser {
    def parse(s: String): JValue = {
      new JValue {
        override def \(nameToFind: String): JValue =
          JArray(List(
            JObject(List(
              JField("name", JString("John Derp")),
              JField("address", JString("Jem Street 21")))),
            JObject(List(
              JField("name", JString("Scala Jo")),
              JField("address", JString("in my sweet dream"))))))
      }
    }
  }

  val json = JsonParser.parse("Your JSON")

Under the hood Scala compiler generates the following:

  val res = (json \ "records")
    .filter(_.isInstanceOf[JObject])
    .flatMap { x =>
      x match {
        case JObject(obj) => //
          obj //
            .withFilter(f => f match {
              case JField("name", _) => true
              case _                 => false
            }) //
            .flatMap(n => obj.withFilter(f => f match {
              case JField("address", _) => true
              case _                    => false
            }).map(a => Map(
              "name" -> (n.value match { case JString(name) => name }),
              "address" -> (a.value match { case JString(address) => address }))))
      }
    }

First line JObject(rec) <- json \ "records" is possible because JArray.filter returns List[JValue] (i.e. List[JObject]). Here each value of List[JValue] maps to JObject(rec) with pattern matching.

Rest calls are series of flatMap and map (this is how Scala for comprehensions work) with pattern matching.

I used Scala 2.11.4.

Of course, match expressions above are implemented using series of type checks and casts.

UPDATE:

When you use Json4s library there is an implicit conversion from JValue to org.json4s.MonadicJValue. See package object json4s:

implicit def jvalue2monadic(jv: JValue) = new MonadicJValue(jv)

This conversion is used here: JObject(rec) <- json \ "records". First, json is converted to MonadicJValue, then def \("records") is applied, then def filter is used on the result of def \ which is JValue, then it is again implicitly converted to MonadicJValue, then def filter of MonadicJValue is used. The result of MonadicJValue.filter is List[JValue]. After that steps described above are performed.

Answer 2

You are using a Scala for comprehension and I believe much of the confusion is about how for comprehensions work. This is Scala syntax for accessing the map, flatMap and filter methods of a monad in a concise way for iterating over collections. You will need some understanding of monads and for comprehensions in order to fully comprehend this. The Scala documentation can help, and so will a search for "scala for comprehension". You will also need to understand about extractors in Scala.

You asked about the meaning of this line:

JObject(rec) <- json \ "records"

This is part of the for comprehension.

Your statement:

I understand that the json \ "records" produces a JArray object,

is slightly incorrect. The \ function extracts a List[JSObject] from the parser result, json

but why is it fetched as JObject(rec) at left of <-?

The json \ "records" uses the json4s extractor \ to select the "records" member of the Json data and yield a List[JObject]. The <- can be read as "is taken from" and implies that you are iterating over the list. The elements of the list have type JObject and the construct JObject(rec) applies an extractor to create a value, rec, that holds the content of the JObject (its fields).

how come it's fetched as JObject(rec) at left of <-?

That is the Scala syntax for iterating over a collection. For example, we could also write:

for (x <- 1 to 10)

which would simply give us the values of 1 through 10 in x. In your example, we're using a similar kind of iteration but over the content of a list of JObjects.

What is the meaning of the JObject(rec)?

This is a Scala extractor. If you look in the json4s code you will find that JObject is defined like this:

case class JObject(obj: List[JField]) extends JValue

When we have a case class in Scala there are two methods defined automatically: apply and unapply. The meaning of JObject(rec) then is to invoke the unapply method and produce a value, rec, that corresponds to the value obj in the JObject constructor (apply method). So, rec will have the type List[JField].

Where does the rec variable come from?

It comes from simply using it and is declared as a placeholder for the obj parameter to JObject's apply method.

Does JObject(rec) mean instantiating new JObject class from rec input?

No, it doesn't. It comes about because the JArray resulting from json \ "records" contains only JObject values.

So, to interpret this:

JObject(rec) <- json \ "records"

we could write the following pseudo-code in english:

Find the "records" in the parsed json as a JArray and iterate over them. The elements of the JArray should be of type JObject. Pull the "obj" field of each JObject as a list of JField and assign it to a value named "rec".

Hopefully that makes all this a bit clearer?

it's also helpful if you can show me the Java equivalent code for the loop above.

That could be done, of course, but it is far more work than I'm willing to contribute here. One thing you could do is compile the code with Scala, find the associated .class files, and decompile them as Java. That might be quite instructive for you to learn how much Scala simplifies programming over Java. :)

why I can't do this? for ( rec <- json \ "records", so rec become JObject. What is the reason of JObject(rec) at the left of <- ?

You could! However, you'd then need to get the contents of the JObject. You could write the for comprehension this way:

val records: List[Map[String, Any]] = for {
    obj: JObject <- json \ "records"
    rec = obj.obj
    JField("name", JString(name)) <- rec
    JField("address", JString(address)) <- rec
  } yield Map("name" -> name, "address" -> address)

It would have the same meaning, but it is longer.

I just want to understand what does the N(x) pattern mean, because I only ever see for (x <- y pattern before.

As explained above, this is an extractor which is simply the use of the unapply method which is automatically created for case classes. A similar thing is done in a case statement in Scala.

UPDATE: The code you provided does not compile for me against 3.2.11 version of json4s-native. This import:

import org.json4s.JsonAST._

is redundant with this import:

import org.json4s._

such that JObject is defined twice. If I remove the JsonAST import then it compiles just fine.

To test this out a little further, I put your code in a scala file like this:

package example

import org.json4s._
// import org.json4s.JsonAST._
import org.json4s.native.JsonParser

class ForComprehension {
  val json = JsonParser.parse(
    """{
      |"records":[
      |{"name":"John Derp","address":"Jem Street 21"},
      |{"name":"Scala Jo","address":"in my sweet dream"}
      |]}""".stripMargin
  )

  val records: List[Map[String, Any]] = for {
    JObject(rec) <- json \ "records"
    JField("name", JString(name)) <- rec
    JField("address", JString(address)) <- rec
  } yield Map("name" -> name, "address" -> address)

  println(records)
}

and then started a Scala REPL session to investigate:

scala> import example.ForComprehension
import example.ForComprehension

scala> val x = new ForComprehension
List(Map(name -> John Derp, address -> Jem Street 21), Map(name -> Scala Jo, address -> in my sweet dream))
x: example.ForComprehension = example.ForComprehension@5f9cbb71

scala> val obj = x.json \ "records"
obj: org.json4s.JValue = JArray(List(JObject(List((name,JString(John Derp)), (address,JString(Jem Street 21)))), JObject(List((name,JString(Scala Jo)), (address,JString(in my sweet dream))))))

scala> for (a <- obj) yield { a }
res1: org.json4s.JValue = JArray(List(JObject(List((name,JString(John Derp)), (address,JString(Jem Street 21)))), JObject(List((name,JString(Scala Jo)), (address,JString(in my sweet dream))))))

scala> import org.json4s.JsonAST.JObject
for ( JObject(rec) <- obj ) yield { rec }
import org.json4s.JsonAST.JObject

scala> res2: List[List[org.json4s.JsonAST.JField]] = List(List((name,JString(John Derp)), (address,JString(Jem Street 21))), List((name,JString(Scala Jo)), (address,JString(in my sweet dream))))

So:

• You are correct, the result of the \ operator is a JArray
• The "iteration" over the JArray just treats the entire array as the only value in the list
• There must be an implicit conversion from JArray to JObject that permits the extractor to yield the contents of JArray as a List[JField].
• Once everything is a List, the for comprehension proceeds as normal.

Hope that helps with your understanding of this.

For more on pattern matching within assignments, try this blog

UPDATE #2: I dug around a little more to discover the implicit conversion at play here. The culprit is the \ operator. To understand how json \ "records" turns into a monadic iterable thing, you have to look at this code:

• org.json4s package object: This line declares an implicit conversion from JValue to MonadicJValue. So what's a MonadicJValue?
• org.json4s.MonadicJValue: This defines all the things that make JValues iterable in a for comprehension: filter, map, flatMap and also provides the \ and \\ XPath-like operators

So, essentially, the use of the \ operator results in the following sequence of actions: - implicitly convert the json (JValue) into MonadicJValue - Apply the \ operator in MonadicJValue to yield a JArray (the "records") - implicitly convert the JArray into MonadicJValue - Use the MonadicJValue.filter and MonadicJValue.map methods to implement the for comprehension

Answer 3

Just simplified example, how for-comprehesion works here:

scala> trait A
defined trait A

scala> case class A2(value: Int) extends A
defined class A2

scala> case class A3(value: Int) extends A
defined class A3

scala> val a = List(1,2,3)
a: List[Int] = List(1, 2, 3)

scala> val a: List[A] = List(A2(1),A3(2),A2(3))
a: List[A] = List(A2(1), A3(2), A2(3))

So here is just:

scala> for(A2(rec) <- a) yield rec //will return and unapply only A2 instances
res34: List[Int] = List(1, 3)

Which is equivalent to:

scala> a.collect{case A2(rec) => rec}
res35: List[Int] = List(1, 3)

Collect is based on filter - so it's enough to have filter method as JValue has.

P.S. There is no foreach in JValue - so this won't work for(rec <- json \ "records") rec. But there is map, so that will: for(rec <- json \ "records") yield rec

If you need your for without pattern matching:

for {
   rec <- (json \ "records").filter(_.isInstanceOf[JObject]).map(_.asInstanceOf[JObject])
   rcobj = rec.obj 
   name <- rcobj if name._1 == "name" 
   address <- rcobj if address._1 == "address" 
   nm = name._2.asInstanceOf[JString].s
   vl = address._2.asInstanceOf[JString].s
} yield Map("name" -> nm, "address" -> vl) 

res27: List[scala.collection.immutable.Map[String,String]] = List(Map(name -> John Derp, address -> Jem Street 21), Map(name -> Scala Jo, address -> in my sweet dream))