Check if table exists in hive metastore using Pyspark

Question

I am trying to check if a table exists in hive metastore if not, create the table. And if the table exists, append data.

I have a snippet of the code below:

spark.catalog.setCurrentDatabase("db_name")
db_catalog = spark.catalog.listTables(dbName = 'table_name)
if any(table_name in row for row in db_catalog):
    add data
else:
    create table

However, I am getting an error.

>>> ValueError: Some of types cannot be determined after inferring

I am unable to resolve the value error as I get the same errors for other databases' tables created in hive metastore. Is there another way to check if table exists in hive metastore?

Answer 1

You can use JVM object for this.

if spark._jsparkSession.catalog().tableExists('db_name', 'tableName'):
  print("exist")
else:
  print("Not Exist")

Py4j socket used for Python functionality. Others operation uses JVM SparkContext.

In Spark Scala you can directly access.

spark.catalog.tableExists("dbName.tableName")