Converting Map type in Case Class to StructField Type

Question

I have a case class which I want to convert to schema in Spark

case class test(request1:Map[String, Any],response1:Option[String] = None,)

How do I convert this class to schema object

val mySchema = StructType(
StructField("request1", Map[String, Any], false),StructField(" response1", Option[String],true))

Map and Options are not available in DataType

Answer 1

It is not possible to use this case class to create a DataFrame schema. While Spark supports map via MapType and Options are handled using wrapped type with Nones converted to NULLs, schema of type Any is not supported.

Assuming you change Value type to String:

case class Test(request1: Map[String, String], response1: Option[String] = None)

corresponding schema should look like this:

StructType(Seq(
   StructField("request1", MapType(StringType, StringType, true), true),
   StructField("response1", StringType, true)
))