How to convert scala.collection.Set to java.util.Set with serializable within an RDD

Question

I have a scala.collection.Set scalaSet : Set[Long].

How will I be able to convert it into a java.util.Set with serializable. I tried the following code, but got java.io.notserializableexception: scala.collection.convert.wrappers$setWrapper

import scala.collection.JavaConversions._

Class MySerializableClass extends Serializable {

    // method to implement the Scala to Java operations on the given RDD
    def rddOps(dummyRDD: RDD[(Long, Set[Long])]) = {
        val dummyRDDWithJavaSet = dummyRDD.map( {
            case(key, value) => (key, scalaToJavaSetConverter(value))
    }

    // scala Set to Java Set Converters
    def scalaToJavaSetConverter(scalaSet: Set[Long]): java.util.Set[Long] = {
        val javaSet : java.util.Set[Long] = setAsJavaSet(scalaSet)
        javaSet
    }
}

I have seen the thread notserializable exception when trying to serialize java map converted from scala for an answer, but the solution didn't work with serialization

Answer 1

The serialization issue with the scala.collection.JavaConvertions/JavaConverters is that these converters are wrappers that use the underlying (scala/java) object. They are merely a wrapper and therefore for it to be effectively serializable, they must have a warranty that the underlying structure is serializable.

The easiest solution in your case is to implement a structural copy in your conversion method:

// scala Set to Java Set Converters
def scalaToJavaSetConverter(scalaSet: Set[Long]): java.util.Set[Long] = {
    val javaSet = new java.util.HashSet[Long]()
    scalaSet.foreach(entry => javaSet.add(entry))
    javaSet
}