Spark SQL HiveContext - saveAsTable creates wrong schema

Question

I try to store a Dataframe to a persistent Hive table in Spark 1.3.0 (PySpark). This is my code:

sc = SparkContext(appName="HiveTest")
hc = HiveContext(sc)
peopleRDD = sc.parallelize(['{"name":"Yin","age":30}'])
peopleDF = hc.jsonRDD(peopleRDD)
peopleDF.printSchema()
#root
# |-- age: long (nullable = true)
# |-- name: string (nullable = true)
peopleDF.saveAsTable("peopleHive")

The Hive output table I expect is:

Column  Data Type   Comments
age     long        from deserializer
name    string      from deserializer

But the actual output Hive table of the above code is:

Column  Data Type       Comments
col     array<string>   from deserializer

Why is the Hive table not the same schema as the DataFrame? How to achieve the expected output?

Answer 1

It's not the schema is wrong. Hive is not able to correctly read table created by Spark, because it doesn't even have the right parquet serde yet. If you do sqlCtx.sql('desc peopleHive').show(), it should show the correct schema. Or you can use the spark-sql client instead of hive. You can also use the create table syntax to create external tables, which works just like Hive, but Spark has much better support for parquet.