Pyspark: Exception: Java gateway process exited before sending the driver its port number

Question

I'm trying to run pyspark on my macbook air. When i try starting it up I get the error:

Exception: Java gateway process exited before sending the driver its port number 

when sc = SparkContext() is being called upon startup. I have tried running the following commands:

./bin/pyspark ./bin/spark-shell export PYSPARK_SUBMIT_ARGS="--master local[2] pyspark-shell" 

with no avail. I have also looked here:

Spark + Python - Java gateway process exited before sending the driver its port number?

but the question has never been answered. Please help! Thanks.

Answer 1

One possible reason is JAVA_HOME is not set because java is not installed.

I encountered the same issue. It says

Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/spark/launcher/Main : Unsupported major.minor version 51.0     at java.lang.ClassLoader.defineClass1(Native Method)     at java.lang.ClassLoader.defineClass(ClassLoader.java:643)     at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)     at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)     at java.net.URLClassLoader.access$000(URLClassLoader.java:73)     at java.net.URLClassLoader$1.run(URLClassLoader.java:212)     at java.security.AccessController.doPrivileged(Native Method)     at java.net.URLClassLoader.findClass(URLClassLoader.java:205)     at java.lang.ClassLoader.loadClass(ClassLoader.java:323)     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:296)     at java.lang.ClassLoader.loadClass(ClassLoader.java:268)     at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:406) Traceback (most recent call last):   File "<string>", line 1, in <module>   File "/opt/spark/python/pyspark/conf.py", line 104, in __init__     SparkContext._ensure_initialized()   File "/opt/spark/python/pyspark/context.py", line 243, in _ensure_initialized     SparkContext._gateway = gateway or launch_gateway()   File "/opt/spark/python/pyspark/java_gateway.py", line 94, in launch_gateway     raise Exception("Java gateway process exited before sending the driver its port number") Exception: Java gateway process exited before sending the driver its port number 

at sc = pyspark.SparkConf(). I solved it by running

sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-java8-installer 

which is from https://www.digitalocean.com/community/tutorials/how-to-install-java-with-apt-get-on-ubuntu-16-04

Answer 2

this should help you

One solution is adding pyspark-shell to the shell environment variable PYSPARK_SUBMIT_ARGS:

export PYSPARK_SUBMIT_ARGS="--master local[2] pyspark-shell" 

There is a change in python/pyspark/java_gateway.py , which requires PYSPARK_SUBMIT_ARGS includes pyspark-shell if a PYSPARK_SUBMIT_ARGS variable is set by a user.