Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Converting Map type in Case Class to StructField Type

I have a case class which I want to convert to schema in Spark

case class test(request1:Map[String, Any],response1:Option[String] = None,)

How do I convert this class to schema object

val mySchema = StructType(
StructField("request1", Map[String, Any], false),StructField(" response1", Option[String],true))

Map and Options are not available in DataType

like image 619
user23577 Avatar asked Jan 06 '16 09:01

user23577


1 Answers

It is not possible to use this case class to create a DataFrame schema. While Spark supports map via MapType and Options are handled using wrapped type with Nones converted to NULLs, schema of type Any is not supported.

Assuming you change Value type to String:

case class Test(request1: Map[String, String], response1: Option[String] = None)

corresponding schema should look like this:

StructType(Seq(
   StructField("request1", MapType(StringType, StringType, true), true),
   StructField("response1", StringType, true)
))
like image 111
zero323 Avatar answered Sep 25 '22 17:09

zero323