Can Spark read data directly into a nested case class?

Question

Say you have a CSV with three columns: item, username, and userid. It is a fairly simple matter to use Spark's Dataset API to read this in:

case class Flat(item: String, username: String, userid: String)
ds = sparkSession.read.csv("path/to/data").toDF("item", "username", "userid").as[Flat]

Then ds will be of type Dataset[Flat].

But suppose you would prefer that your data have the form Dataset[Nested] where Nested is given by:

case class User(name: String, id: String)
case class Nested(item: String, user: User)

One way to do it is to read the data into a Dataset[Flat] and then apply a map to transform it into a Dataset[Nested], but in practice the Flat case class often isn't needed for anything else and it makes the code unnecessarily verbose. Is there any way to skip the middleman and directly construct a Dataset[Nested]?

Answer 1

Is there any way to skip the middleman and directly construct a Dataset[Nested]?

There is not - Datasets are match by structure and names. You cannot just have names and data has to be reshaped.

If you prefer to skip Flat definition just use dynamic API

import org.apache.spark.sql.functions._

ds.select($"item", struct($"name", $"id") as "user").as[Nested]

as[Flat] doesn't really type check so you don't loose anything.