Workaround for Scala RDD not being covariant

Question

I'm trying to write a function to operate on RDD[Seq[String]] objects, e.g.:

def foo(rdd: RDD[Seq[String]]) = { println("hi") }

This function cannot be called on objects of type RDD[Array[String]]:

val testRdd : RDD[Array[String]] = sc.textFile("somefile").map(_.split("\\|", -1))
foo(testRdd)

->
error: type mismatch;
found   : org.apache.spark.rdd.RDD[Array[String]]
required: org.apache.spark.rdd.RDD[Seq[String]]

I guess that's because RDD isn't covariant.

I've tried a bunch of definitions of foo to get around this. Only one of them has compiled:

def foo2[T[String] <: Seq[String]](rdd: RDD[T[String]]) = { println("hi") }

But it's still broken:

foo2(testRdd)


->
<console>:101: error: inferred type arguments [Array] do not conform to method foo2's type
parameter bounds [T[String] <: Seq[String]]
          foo2(testRdd)
          ^
<console>:101: error: type mismatch;
found   : org.apache.spark.rdd.RDD[Array[String]]
required: org.apache.spark.rdd.RDD[T[String]]

Any idea how I can work around this? This is all taking place in the Spark shell.

Answer 1

For this you can use a view bound.

Array is not a Seq, but it can be viewed as a Seq.

def foo[T <% Seq[String]](rdd: RDD[T]) = ???

The <% says that T can be viewed as a Seq[String] so that whenever you use a Seq[String] method on T then T will be converted to Seq[String].

For Array[A] to be viewed as Seq[A] there needs to be an implicit function in scope that can convert Arrays to Seqs. As Ionuț G. Stan said, it exists in scala.Predef.