Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Spark structured streaming: converting row to json

I'm trying to convert Row of DataFrame into json string using only spark API.

From input Row

+----------------+-----------+
|       someThing|       else|
+----------------+-----------+
|            life|         42|
+----------------+-----------+

with

myDataFrame
.select(struct("*").as("col"))
.select(to_json(col("col")))
.writeStream()
.foreach(new KafkaWriter())
.start()

using KafkaWriter, that is using row.toString() i got:

[{  
    "someThing":"life",
    "else":42
}]

When i would like to get this instead:

{  
    "someThing":"life",
    "else":42
}

(without the [])

Any idea?

like image 367
Martin Brisiak Avatar asked Oct 16 '17 12:10

Martin Brisiak


1 Answers

Just found the solution. Using Row.mkString instead of Row.toString solved my case.

like image 59
Martin Brisiak Avatar answered Oct 24 '22 09:10

Martin Brisiak