Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to parse JSON data in Scala?

I am new to Scala. I want to parse JSON data in scala.

I want to loop this data and in each iteration extract data for id,v,q, and t from values

I am using below code to parse it to JSON

import scala.util.parsing.json._

val data =
  """
{
  "timestamp":
  1518501114949
  , "values":
  [
  {
    "id":
    "abc"
    , "v":
    0
    , "q":
    true
    , "t":
    1518501114487
  }
  ,
  {
    "id":
    "xyz"
    , "v":
    15
    , "q":
    true
    , "t":
    1518501114494
  }
  ]
}
"""

val parsed = JSON.parseFull(data)

I am getting output as below

 Some(Map(timestamp -> 1.518501114949E12, values -> List(Map(id -> abc, v -> 0.0, q -> true, t -> 1.518501114487E12), Map(id -> xyz, v -> 15.0, q -> true, t -> 1.518501114494E12), Map(id -> klm, v -> 12.6999998, q -> true, t -> 1.518501114487E12), Map(id -> 901.Hotmelt.PSA.0759_PSAM01_Vac, v -> 1.0, q -> true, t -> 1.518501114494E12))))

but I don't know how to loop and fetch all values after that

and I am not understanding why timestamp is getting converted to E12 values

like image 668
Tarun Khaneja Avatar asked Feb 13 '18 11:02

Tarun Khaneja


People also ask

What is the best JSON library for Scala?

Argonaut is a great library. It's by far the best JSON library for Scala, and the best JSON library on the JVM. If you're doing anything with JSON in Scala, you should be using Argonaut. circe is a fork of Argonaut with a few important differences.

What is Circe Scala?

Circe is a Scala library that simplifies working with JSON, allowing us to easily decode a JSON string into a Scala object or convert a Scala object to JSON. The library automatically generates the object encoders and decoders, thereby reducing the lines of code we need to work with JSON in Scala.

How does spark read JSON?

Spark SQL can automatically infer the schema of a JSON dataset and load it as a Dataset<Row> . This conversion can be done using SparkSession. read(). json() on either a Dataset<String> , or a JSON file.


2 Answers

The problem is that the parseFull returns an Option with an Any inside, so you first need to get rid of that:

With this code below, you will keep the values:

val listAsAny = parsed match {
  case Some(e:Map[Any,Any]) => e("values")
  case None => println("Failed.")
}

But they still as Any, so you can transform it as follows:

val values = listAsAny.asInstanceOf[List[Map[String, Any]]]

Now values is a List of maps with the following values, and you can get the values inside as you will do with a regular List

List(Map(id -> abc, v -> 0.0, q -> true, t -> 1.518501114487E12), Map(id -> xyz, v -> 15.0, q -> true, t -> 1.518501114494E12))

For instance, to retrieve the ids you can do:

values.map(_("id"))

And the result will be:

List(abc, xyz)
like image 51
SCouto Avatar answered Oct 14 '22 15:10

SCouto


The upickle library allows for a robust, elegant solution.

val parsed = ujson.read(data)
parsed("values").arr.map(_("id").str) // ArrayBuffer("abc", "xyz")

See here for a more detailed discussion on why upickle / ujson is the best Scala library for parsing JSON.

like image 35
Powers Avatar answered Oct 14 '22 15:10

Powers