message:Hive Schema version 1.2.0 does not match metastore's schema version 2.1.0 Metastore is not upgraded or corrupt

Question

enviroment: spark2.11 hive2.2 hadoop2.8.2

hive shell run successfully! and hava no error or warning. but when run application.sh, start failed

/usr/local/spark/bin/spark-submit \
--class cn.spark.sql.Demo \
--num-executors 3 \
--driver-memory 512m \
--executor-memory 512m \
--executor-cores 3 \
--files /usr/local/hive/conf/hive-site.xml \
--driver-class-path /usr/local/hive/lib/mysql-connector-java.jar \
/usr/local/java/sql/sparkstudyjava.jar \

and the error tips:

Exception in thread "main" java.lang.IllegalArgumentException: Error while 
instantiating 'org.apache.spark.sql.hive.HiveSessionState':
...
Caused by: java.lang.IllegalArgumentException: Error while instantiating 
'org.apache.spark.sql.hive.HiveExternalCatalog':
...
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Unable to 
instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
Caused by: java.lang.RuntimeException: Unable to instantiate 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
Caused by: MetaException(message:Hive Schema version 1.2.0 does not match 
metastore's schema version 2.1.0 Metastore is not upgraded or corrupt)
...

i try many method to solving this errors, but errors still occurs. how to fix?

Answer 1

Probably hive is referring to another version of hive (which is configured differently). Execute below command & see if output is different from /usr/local/hive.

$which hive

If both are same hive directories, add below properties in hive-site.xml.

<property>
    <name>hive.metastore.schema.verification</name>
    <value>false</value>
</property>

Answer 2

Sometimes the hive jar in spark could be different than hive version installed, to handle this scenario, You can Pass the jars and version in conf parameter to spark job submit E.g. --conf spark.sql.hive.metastore.version=2.3.0 --conf spark.sql.hive.metastore.jars=/home/apache-hive-2.3.6-bin/lib/*