How to fix spark-shell on Windows (fails with "was unexpected at this time")? [closed]

Question

I am getting following error when running spark-shell command:

I have downloaded spark-2.1.1-bin-hadoop2.7.tgz file from http://spark.apache.org/downloads.html, extracted the tar file and pasted the contents of the folder into c:\Spark directory.

After that I have configured the environment variable for spark and jdk accordingly, but I am getting this error.

Answer 1

I resolved this error by using a short hand like "Progra~1" which is a short hand for "Program Files (x86)".

C:\Progra~1\Java\jdk1.8.0_161

Answer 2

I'm almost sure that your JAVA_HOME environment variable contains a space that breaks spark-shell. Please re-install Java to a directory with no spaces in the path.

---

You can see the relevant piece of code in bin/spark-class2.cmd that spark-shell executes on Windows under the covers (through bin/spark-submit2.cmd shell script):

if "x%1"=="x" (

So when spark-class2.cmd substitutes %1 to a path with a space (or something similar) it ends up as:

if "x"Files\Java\jdk1.8.0_45""=="x" (

that gives the error due to too many double quotes.

The mystery is how does the JAVA_HOME end up in this place. I can't seem to find the reason, but that's what we see here.

Answer 3

You can see this issue when your Java home set like following in windows

JAVA_HOME=C:\Program Files (x86)\Java\jdk1.8.0_162\bin

Issue here is space in "Program Files (x86)" If you add double quotes will not work on window 10

"C:\Program Files (x86)\Java\jdk1.8.0_162\bin" 

You need to copy Java into outside Program Files (x86) then it should work

JAVA_HOME=C:\java\jdk1.8.0_171\bin