Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout does not exist in the JVM

Question

I am trying to create SparkContext in jupyter notebook but I am getting following Error:

Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout does not exist in the JVM

Here is my code

from pyspark import SparkContext, SparkConf
conf = SparkConf().setMaster("local").setAppName("Groceries")
sc = SparkContext(conf = conf)


Py4JError                                 Traceback (most recent call last)
<ipython-input-20-5058f350f58a> in <module>
      1 conf = SparkConf().setMaster("local").setAppName("My App")
----> 2 sc = SparkContext(conf = conf)

~/Documents/python38env/lib/python3.8/site-packages/pyspark/context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    144         SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
    145         try:
--> 146             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
    147                           conf, jsc, profiler_cls)
    148         except:

~/Documents/python38env/lib/python3.8/site-packages/pyspark/context.py in _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, jsc, profiler_cls)
    224         self._encryption_enabled = self._jvm.PythonUtils.isEncryptionEnabled(self._jsc)
    225         os.environ["SPARK_AUTH_SOCKET_TIMEOUT"] = \
--> 226             str(self._jvm.PythonUtils.getPythonAuthSocketTimeout(self._jsc))
    227         os.environ["SPARK_BUFFER_SIZE"] = \
    228             str(self._jvm.PythonUtils.getSparkBufferSize(self._jsc))

~/Documents/python38env/lib/python3.8/site-packages/py4j/java_gateway.py in __getattr__(self, name)
   1528                     answer, self._gateway_client, self._fqn, name)
   1529         else:
-> 1530             raise Py4JError(
   1531                 "{0}.{1} does not exist in the JVM".format(self._fqn, name))
   1532 

Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout does not exist in the JVM

Answer 1

Python's pyspark and spark cluster versions are inconsistent and this error is reported. Uninstall the version that is consistent with the current pyspark, then install the same version as the spark cluster. My spark version is 3.0.2 and run the following code:

pip3 uninstall pyspark
pip3 install pyspark==3.0.2