Spark-Shell Startup Errors

Question

I am seeing errors when starting spark-shell, using spark-1.6.0-bin-hadoop2.6. This is new behavior that just arose.

The upshot of the failures displayed in the log messages below, is that sqlContext is not available (but sc is).

Is there some kind of Derby lock that could be released? Another instance of Derby may have already booted the database /root/spark-1.6.0-bin-hadoop2.6/bin/metastore_db.

<console>:16: error: not found: value sqlContext
         import sqlContext.implicits._
                ^
<console>:16: error: not found: value sqlContext
         import sqlContext.sql

16/05/25 11:00:00 ERROR Schema: Failed initialising database.
Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@c2191a8, see the next exception for details.
org.datanucleus.exceptions.NucleusDataStoreException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@c2191a8, see the next exception for details.


16/05/25 11:06:02 WARN Hive: Failed to access metastore. This class should not accessed in runtime.
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient


16/05/25 11:06:02 ERROR Schema: Failed initialising database.
Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@372e972d, see the next exception for details.
org.datanucleus.exceptions.NucleusDataStoreException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@372e972d, see the next exception for details.

Caused by: java.sql.SQLException: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@c2191a8, see the next exception for details.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        ... 134 more
Caused by: java.sql.SQLException: Another instance of Derby may have already booted the database /root/spark-1.6.0-bin-hadoop2.6/bin/metastore_db.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
        ... 131 more
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /root/spark-1.6.0-bin-hadoop2.6/bin/metastore_db.

Answer 1

I had a similar problem in a Spark 2.0.0 shell when trying to create a DataFrame, just remove metastore_db/dbex.lck and the problem is fixed.

Answer 2

similar to Andy's answer. I had the same issue in windows, and here is the solution:

1. Run cmd on windows as administrator
2. navigate to spark home directory

3. open spark-shell

   c:\park\bin> spark-shell