Why does spark-shell fail with “error: not found: value spark”?

Question

I use Spark 2.1.0.

When I run spark-shell, I encounter this error:

<console>:14: error: not found: value spark
       import spark.implicits._
              ^
<console>:14: error: not found: value spark
       import spark.sql
              ^

What could be the reason? How to fix it?

Answer 1

I was facing the same issue, after investigation i observed there was the compatibility issue between spark version and winutils.exe of hadoop-2.x.x.

After experiment i suggest you to use hadoop-2.7.1 winutils.exe with spark-2.2.0-bin-hadoop2.7 version and hadoop-2.6.0 winutils.exe with spark-1.6.0-bin-hadoop2.6 version and set below environment variables

SCALA_HOME  : C:\Program Files (x86)\scala2.11.7;
JAVA_HOME   : C:\Program Files\Java\jdk1.8.0_51
HADOOP_HOME : C:\Hadoop\winutils-master\hadoop-2.7.1
SPARK_HOME  : C:\Hadoop\spark-2.2.0-bin-hadoop2.7
PATH    : %JAVA_HOME%\bin;%SCALA_HOME%\bin;%HADOOP_HOME%\bin;%SPARK_HOME%\bin;

Create C:\tmp\hive diroctory and give access permission using below command

C:\Hadoop\winutils-master\hadoop-2.7.1\bin>winutils.exe chmod -R 777 C:\tmp\hive

Remove local Derby-based metastore metastore_db directory from Computer if it exist.

C:\Users\<User_Name>\metastore_db

Use below command to start spark shell

C:>spark-shell

Answer 2

The reason for the error is that the instance could not be created due to some earlier issues (which may have happened because you are on Windows and you have not installed winutils.exe binary or some other session keeps the local Derby-based metastore).

The recommendation is to scroll up and review the entire screen of logs where you find the root cause.