java.io.InvalidClassException: org.apache.spark.internal.io.HadoopMapReduceCommitProtocol; local class incompatible

Question

CONTEXT

I am executing a Spark tool (if interested, this is the Spark tool) on a Spark cluster. The tool is reading an input file from HDFS and will generate an output file in HDFS.

I have 2 Azure VM, with a Spark Master container, a Namenode container, two Spark Worker containers and two Datanode containers (more two containers to provide a file required by the tool, but I don't thin it's important), provisioned with Docker Swarm.

This is the Bash command that I use to run the Spark tool:

/gatk/gatk BwaAndMarkDuplicatesPipelineSpark \
--input hdfs://namenode:8020/PFC_0028_SW_CGTACG_R_fastqtosam.bam \
--reference hdfs://namenode:8020/hg19-ucsc/ucsc.hg19.2bit \
--bwa-mem-index-image /reference_image/ucsc.hg19.fasta.img \
--disable-sequence-dictionary-validation true \
--output hdfs://namenode:8020/PFC_0028_SW_CGTACG_R_dedup_reads.bam \
-- --spark-runner SPARK --spark-master spark://$SPARK_MASTER_HOST:7077 \
--driver-memory 30g --executor-cores 4 --executor-memory 15g

PROBLEM

After some hours I receive this error:

18/02/08 23:10:42 WARN TaskSetManager: Lost task 15.0 in stage 5.0 (TID 3087, 10.0.0.10, executor 0): 
java.io.InvalidClassException: org.apache.spark.internal.io.HadoopMapReduceCommitProtocol; 
local class incompatible: stream classdesc serialVersionUID = -3240062154626659006, local class serialVersionUID = 8531775026770993759

Searching even here on Stack Overflow I found this issue with similar Exception, it basically suggests to add something like this

   private static final long serialVersionUID = 6529685098267757690L;

in Java code. But the Spark tool that I am using, is actually using the serialVerionUID, moreover the exception suggests that is raised by org.apache.spark.internal.io.HadoopMapReduceCommitProtocol, which seems to extend Serializable, but doesn't use a serialVerionUID; am I supposed to open an issue on Spark GitHub and report this exception or am I doing something wrong?

Sorry if too verbose, if necessary more details feel free to ask.

Answer 1

I tried to inspect the Spark version of Spark Master container and Spark Worker container and it was respectively 2.2.1 and 2.2.0

So I used the 2.2.0 in both of them and this time it worked. I assume that the problem was the mismatching version between Worker and Master (even if the first time that I launched the tool the exception was raised only after some hours of execution).