Spark does't run in Windows anymore

Question

I have Windows 10 and I followed this guide to install Spark and make it work on my OS, as long as using Jupyter Notebook tool. I used this command to instantiate the master and import the packages I needed for my job: pyspark --packages graphframes:graphframes:0.8.1-spark3.0-s_2.12 --master local[2]

However, later, I figured out that any worker wasn't instantiated according to the aforementioned guide and my tasks were really slow. Therefore, taking inspiration from this, since I could not find any other way to connect workers to the Cluster manager due to the fact it was run by Docker, I tried to set up everything manually with the following commands: bin\spark-class org.apache.spark.deploy.master.Master

The master was correctly instantiated, so I continued by the next command: bin\spark-class org.apache.spark.deploy.worker.Worker spark://<master_ip>:<port> --host <IP_ADDR> Which returned me the following error:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/04/01 14:14:21 INFO Master: Started daemon with process name: 8168@DESKTOP-A7EPMQG
21/04/01 14:14:21 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[main,5,main]
java.lang.ExceptionInInitializerError
        at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit>(ByteArrayMethods.java:54)
        at org.apache.spark.internal.config.package$.<init>(package.scala:1006)
        at org.apache.spark.internal.config.package$.<clinit>(package.scala)
        at org.apache.spark.deploy.master.MasterArguments.<init>(MasterArguments.scala:57)
        at org.apache.spark.deploy.master.Master$.main(Master.scala:1123)
        at org.apache.spark.deploy.master.Master.main(Master.scala)
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @60015ef5
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357)
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
        at java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:188)
        at java.base/java.lang.reflect.Constructor.setAccessible(Constructor.java:181)
        at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:56)
        ... 6 more

From that moment on, none of the commands I used to run before were working anymore, and they returned the message you can see. I guess I messed up some Java stuff, but I do not understand what and where, honestly.

My java version is:

java version "16" 2021-03-16
Java(TM) SE Runtime Environment (build 16+36-2231)
Java HotSpot(TM) 64-Bit Server VM (build 16+36-2231, mixed mode, sharing)

Answer 1

I got the same error just now, the issue seems with Java version.

1. I installed java, python, spark etc. All latest versions... !
2. Followed steps mentioned in the below link..

https://phoenixnap.com/kb/install-spark-on-windows-10

3. Got the same error as you.. !
4. Downloaded Java SE 8 version from Oracle site..

https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.html

Downloaded jdk-8u281-windows-x64.exe

5. Reset the JAVA_HOME.
6. Started spark-shell - it opened perfectly without any issues.

FYI: I don't have neither java or spark experience, if anyone feels something is wrong please correct me. Just that it worked for me, so providing the same solution here.. :)

Thanks, Karun

Answer 2

I got a similar error on MacOS. The problem was with Java (I was using JDK 17), had to downgrade or use a different version.

Ended up using this: https://adoptium.net/releases.html?variant=openjdk11 Download and install. Might have to remove your JDK17 version.