Spark job did not find table in Hive database

Question

I have a table in default database of Hive, and can sucessfully get records from that table in command line:

>hive: select * from my_table;

But when I create a job in Spark to run, it just throwns me exception like this:

INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=my_table
16/01/04 03:41:42 INFO HiveMetaStore.audit: ugi=etl     ip=unknown-ip-addr      cmd=get_table : db=default tbl=my_table
Exception in thread "main" org.apache.spark.sql.AnalysisException: no such table my_table;

Here is the code of that job:

SparkConf conf = new SparkConf().setMaster("local").setAppName("PhucdullApp");
        JavaSparkContext sc = new JavaSparkContext(conf);
        HiveContext sqlContext = new org.apache.spark.sql.hive.HiveContext(sc.sc());
        DataFrame df = sqlContext.sql("select * from my_table");
        df.show();
        sc.close();

Answer 1

(Assuming that you are using default Derby database which is configured in hive-default.xml file). Ensure that you have followed the following steps: -

1. Copy hive-default.xml from $HIVE_HOME/conf/ to $SPARK_HOME/conf/ directory.
2. locate your "metastore_db" directory and submit your Spark Job from the same directory using spark-submit command.