Spark: select with key in map

Question

I need to select with the key geo.cc in a map in a DataFrame:

 |-- params: map (nullable = true)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = true)

but the value of the key, with its dot in the middle, seems to confuse Spark. If I write:

X.filter(X("params.geo.cc") === "us")

I get the error:

org.apache.spark.sql.AnalysisException: Can't extract value from params#3[geo];

What can I do? (needless to say, I do not control the key, i.e. I cannot change that geo.cc string to e.g. geo_cc.

Answer 1

You should use apply:

val df = Seq((1L, Map("geo.cc" -> "US"))).toDF("id", "params")

df.select($"params"("geo.cc") === "US").show
// +-----------------------+
// |(params['geo.cc'] = US)|
// +-----------------------+
// |                   true|
// +-----------------------+

or getItem

df.select($"params".getItem("geo.cc") === "US").show
// +-----------------------+
// |(params['geo.cc'] = US)|
// +-----------------------+
// |                   true|
// +-----------------------+

on a specific column, not DataFrame.