no valid constructor on spark

Question

This is my code:

class FNNode(val name: String)

case class Ingredient(override val name: String, category: String) extends FNNode(name)


val ingredients: RDD[(VertexId, FNNode)] = 
sc.textFile(PATH+"ingr_info.tsv").
      filter(! _.startsWith("#")).
      map(line => line.split('\t')).
      map(x => (x(0).toInt ,Ingredient(x(1), x(2))))

and there are no errors when I define these variables. However, when trying to execute it:

ingredients.take(1)

I get

org.apache.spark.SparkException: Job aborted due to stage failure: Exception while getting task result: java.io.InvalidClassException: $iwC$$iwC$Ingredient; no valid constructor
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)

It seems this could be related to Serialization issues as per the answer here . However, I have no idea of how to solve this if it is indeed a Serialization issue.

I'm following along the code in this book by they way, so I would assume this should have at least worked at some point?

Answer 1

This fixed your issue for me:

class FNNode(val name: String) extends Serializable