Table loaded through Spark not accessible in Hive

Question

Hive table created through Spark (pyspark) are not accessible from Hive.

df.write.format("orc").mode("overwrite").saveAsTable("db.table")

Error while accessing from Hive:

Error: java.io.IOException: java.lang.IllegalArgumentException: bucketId out of range: -1 (state=,code=0)

Table getting created successfully in Hive and able to read this table back in spark. Table metadata is accessible (in Hive) and data file in table (in hdfs) directory.

TBLPROPERTIES of Hive table are :

  'bucketing_version'='2',                         
  'spark.sql.create.version'='2.3.1.3.0.0.0-1634', 
  'spark.sql.sources.provider'='orc',              
  'spark.sql.sources.schema.numParts'='1',

I also tried creating table with other workarounds but getting error while creating table:

df.write.mode("overwrite").saveAsTable("db.table")

OR

df.createOrReplaceTempView("dfTable")
spark.sql("CREATE TABLE db.table AS SELECT * FROM dfTable")

Error :

AnalysisException: u'org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Table default.src failed strict managed table checks due to the following reason: Table is marked as a managed table but is not transactional.);'

Stack version details:

Spark2.3

Hive3.1

Hortonworks Data Platform HDP3.0

Answer 1

From HDP 3.0, catalogs for Apache Hive and Apache Spark are separated, and they use their own catalog; namely, they are mutually exclusive - Apache Hive catalog can only be accessed by Apache Hive or this library, and Apache Spark catalog can only be accessed by existing APIs in Apache Spark . In other words, some features such as ACID tables or Apache Ranger with Apache Hive table are only available via this library in Apache Spark. Those tables in Hive should not directly be accessible within Apache Spark APIs themselves.

• Below article explain the steps:

Integrating Apache Hive with Apache Spark - Hive Warehouse Connector