How to pattern match on Row with null values?

Question

I'm wondering why I cant pattern match on a Spark (2.1) Row containing null values (Strings):

val r = Row(null:String)

r match {case Row(s:String) => println("row is null")}

scala.MatchError: [null] (of class org.apache.spark.sql.catalyst.expressions.GenericRow)

Answer 1

Row doesn't really play an important role here, we can simplify:

val r = null: String

r match {case s:String => println("s is null")}

You can check the pattern match still fails. This is because type patterns like s: String are specifically defined not to match null:

A type pattern T is of one of the following forms:

• A reference to a class C, p.C, or T#C. This type pattern matches any non-null instance of the given class...

isInstanceOf behaves like this too: null.isInstanceOf[String] is false.

So if you want to match null, you can

1) use exactly null as the pattern:

r match {
  case s: String => println("r is not null")
  case null => println("r is null")
}

2) use a "catch-all" pattern like _ or a variable:

r match {
  case s: String => println("r is not null")
  case s => println("matches only if r is null or not a String")
}

Or if we put Row back, you'd write

r match {
  case Row(s: String) => println("r contains non-null")
  case Row(null) => println("r contains null")
}