local class incompatible Exception: when running spark standalone from IDE

Question

I begin to test spark. I installed spark on my local machine and run a local cluster with a single worker. when I tried to execute my job from my IDE by setting the sparconf as follows:

final SparkConf conf = new SparkConf().setAppName("testSparkfromJava").setMaster("spark://XXXXXXXXXX:7077");
final JavaSparkContext sc = new JavaSparkContext(conf);
final JavaRDD<String> distFile = sc.textFile(Paths.get("").toAbsolutePath().toString() + "dataSpark/datastores.json");*

I got this exception:

java.lang.RuntimeException: java.io.InvalidClassException: org.apache.spark.rpc.netty.RequestMessage; local class incompatible: stream classdesc serialVersionUID = -5447855329526097695, local class serialVersionUID = -2221986757032131007

Answer 1

It can be multiple incompatible reasons below:

• Hadoop version;
• Spark version;
• Scala version;
• ...

For me, its Scala version , I using 2.11.X in my IDE but official doc says:

Spark runs on Java 7+, Python 2.6+ and R 3.1+. For the Scala API, Spark 1.6.1 uses Scala 2.10. You will need to use a compatible Scala version (2.10.x).

and the x in the doc told cannot be smaller than 3 if you using latest Java(1.8), cause this. Hope it will help you!

Answer 2

Got it all working with below combination of versions

Installed spark 1.6.2

verify with bin/spark-submit --version

<dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>1.6.2</version>
</dependency>

and

Scala 2.10.6 and Java 8.

Note it did NOT work and have similar class incompatible issue with below versions

Scala 2.11.8 and Java 8

<dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>1.6.2</version>
</dependency>