java.lang.IllegalArgumentException: Too large frame: 5211883372140375593

Question

I submitted my code to the cluster to run, but I encountered the following error.

''' java.lang.IllegalArgumentException: Too large frame: 5211883372140375593 at org.sparkproject.guava.base.Preconditions.checkArgument(Preconditions.java:119) at org.apache.spark.network.util.TransportFrameDecoder.decodeNext(TransportFrameDecoder.java:148)

'''

and my submit command is like this

spark-submit
--master spark://172.16.244.8:6066
--deploy-mode cluster
--num-executors 3
--executor-cores 8
--executor-memory 16g
--driver-memory 2g
--conf spark.default.parallelism=10
--class ParallelComputing
hdfs://172.16.244.5:9000/qlt/portrait-batch-0.0.1-jar-with-dependencies.jar

what is the reason

Answer 1

The reason is because the version of spark does not match the version of the cluster, which can be solved by modifying the local spark version to be consistent with the cluster.